Troubleshooting Kubernetes PVC Stuck in Pending on Ubuntu 20.04 LTS

Resolve Kubernetes Persistent Volume Claims (PVCs) stuck in pending status on Ubuntu 20.04. This guide covers common causes, including missing PVs, StorageClass misconfigurations, and provisioner issues.

When deploying stateful applications in Kubernetes, Persistent Volume Claims (PVCs) are crucial for requesting storage resources. However, it's a common and often frustrating scenario for a PVC to remain in a Pending state, preventing your applications from starting or functioning correctly. This comprehensive guide will walk you through diagnosing and resolving a PVC stuck in Pending status on an Ubuntu 20.04 LTS Kubernetes cluster, leveraging our 16 years of web hosting and DevOps expertise.

Symptom & Error Signature

The primary symptom is that your Kubernetes PVC never transitions from Pending to Bound. When you inspect the PVC, you'll see its status as Pending, and its Volume field will be empty. Further investigation into Kubernetes events will often reveal the underlying reasons.

Example of PVC status:

kubectl get pvc -n my-app-namespace

NAME                STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS         AGE
my-app-data-pvc     Pending                                      standard-storage     5m

Example of events on the PVC:

kubectl describe pvc my-app-data-pvc -n my-app-namespace

Name:          my-app-data-pvc
Namespace:     my-app-namespace
StorageClass:  standard-storage
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-class: standard-storage
               volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/no-provisioner
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
DataSource:    <none>
Events:
  Type     Reason              Age                  From                         Message
  ----     ------              ----                 ----                         -------
  Warning  ProvisioningFailed  4m43s (x10 over 5m)  persistentvolume-controller  Failed to provision volume with StorageClass "standard-storage": storageclass.storage.k8s.io "standard-storage" not found
  Warning  ProvisioningFailed  4m43s (x10 over 5m)  persistentvolume-controller  Failed to provision volume with StorageClass "standard-storage": waiting for a volume to be created, either by a user or by a provisioner

Or, a slightly different message indicating no matching PV:

Events:
  Type     Reason                Age                  From                         Message
  ----     ------                ----                 ----                         -------
  Warning  ProvisioningFailed    3m (x5 over 4m)      persistentvolume-controller  no persistent volumes available for this claim and no storage class is set
  Warning  FailedBinding         3m (x5 over 4m)      persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

Root Cause Analysis

A PVC gets stuck in Pending status primarily because Kubernetes cannot find or provision an appropriate Persistent Volume (PV) to fulfill the claim's requirements. The underlying reasons for this failure are typically one or more of the following:

1. No Matching Persistent Volume (PV) Available (Manual Provisioning): If your StorageClass is configured for manual provisioning (provisioner: kubernetes.io/no-provisioner or volumeBindingMode: WaitForFirstConsumer with no dynamic provisioner), or if you don't have a StorageClass defined, Kubernetes expects pre-created PVs to bind to PVCs. If no PV exists that matches the PVC's storageClassName, accessModes, and capacity requirements, the PVC will remain pending.
2. Missing or Misconfigured StorageClass:
   • The storageClassName specified in the PVC does not exist.
   • The StorageClass exists but is misconfigured (e.g., incorrect provisioner, missing parameters, volumeBindingMode issues).
   • The provisioner defined in the StorageClass is not running or is not correctly installed/configured in the cluster (e.g., the CSI driver pod crashed, NFS provisioner not deployed).
3. Dynamic Provisioner Issues:
   • The chosen storage provisioner (e.g., nfs-subdir-external-provisioner, local-path-provisioner, a cloud-specific CSI driver like aws-ebs-csi-driver, gcp-pd-csi-driver) is not deployed, not running, or is failing to provision volumes due to permissions, connectivity, or quota issues with the underlying storage backend.
   • The provisioner's service account lacks necessary RBAC permissions.
4. Insufficient Storage Capacity: Even if a provisioner is working, it might fail if the underlying storage system (NFS server, cloud disk quota, local disk space) has run out of capacity or cannot fulfill the requested size.
5. Access Modes Mismatch: The PVC requests an accessMode (e.g., ReadWriteOnce, ReadOnlyMany, ReadWriteMany) that no available PV or the configured provisioner can satisfy. For instance, many cloud block storage providers only support ReadWriteOnce.
6. Node Affinity/Taints/Tolerations: If volumeBindingMode: WaitForFirstConsumer is set, the PVC binding is delayed until a Pod using the PVC is scheduled. If that Pod cannot be scheduled due to node affinity, taints, or resource constraints, the PVC will appear stuck. This is more of a Pod scheduling issue that indirectly affects PVC binding.
7. Network or Firewall Issues: The Kubernetes node or the storage provisioner pod cannot reach the external storage backend (e.g., NFS server, iSCSI target, cloud API endpoint) due to network segmentation or firewall rules.

Step-by-Step Resolution

Follow these steps to diagnose and resolve a PVC stuck in Pending. We'll proceed from general checks to more specific provisioner-related issues.

1. Inspect the PVC and its Events

Always start by thoroughly examining the PVC and its associated events. This is the single most important diagnostic step.

# Replace 'my-app-data-pvc' and 'my-app-namespace' with your specific values
kubectl describe pvc my-app-data-pvc -n my-app-namespace

Look for Warning or Failed events in the Events section. These messages will often directly point to the problem, such as "storageclass not found," "no persistent volumes available," or "provisioning failed" with a specific reason.

# For a broader view of events related to storage in the namespace
kubectl get events -n my-app-namespace --field-selector involvedObject.kind=PersistentVolumeClaim

2. Verify the StorageClass

If the events indicate issues with the StorageClass, investigate it next.

# Get all StorageClasses
kubectl get storageclass

NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
standard-storage     kubernetes.io/no-provisioner    Delete          Immediate              false                  2y
nfs-client           cluster.local/nfs-client    Delete          Immediate              true                   1y
local-path (default)  rancher.io/local-path   Delete          WaitForFirstConsumer   false                  6m

Confirm that the STORAGECLASS named in your PVC (e.g., standard-storage from the example) actually exists. If it doesn't, you need to either:

• Create the missing StorageClass.
• Modify your PVC to use an existing StorageClass.
• Remove storageClassName from your PVC YAML to use the default StorageClass (if one is defined).

Inspect the StorageClass definition:

kubectl get storageclass <your-storageclass-name> -o yaml

# Example: Default local-path StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: local-path
provisioner: rancher.io/local-path
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

• provisioner field:
  • If it's kubernetes.io/no-provisioner, it means manual provisioning. Proceed to Step 3.
  • If it specifies a dynamic provisioner (e.g., cluster.local/nfs-client, rancher.io/local-path, ebs.csi.aws.com), ensure that provisioner is actually deployed and running correctly. Proceed to Step 4.
• volumeBindingMode:
  • Immediate: PVC binds as soon as a suitable PV is found/provisioned, regardless of Pod scheduling.
  • WaitForFirstConsumer: Binding is delayed until a Pod using the PVC is scheduled. This helps with topology awareness (e.g., ensuring a local volume is provisioned on the node where the Pod will run). If your Pod is stuck pending, the PVC will also be pending. Check Pod events (kubectl describe pod <pod-name> -n <namespace>) in this case.

3. For Manual Provisioning (kubernetes.io/no-provisioner)

If your StorageClass uses kubernetes.io/no-provisioner or your PVC has no storageClassName, Kubernetes expects you to manually create a Persistent Volume (PV) that matches the PVC's requirements.

Check for existing PVs:

kubectl get pv

Look for PVs that:

• Have Status: Available.
• Match the Capacity requested by the PVC.
• Match the Access Modes (e.g., RWO, ROX, RWX).
• Match the StorageClass name (or lack thereof if PVC has no storageClassName).

Example of a PV definition to match a PVC:

# my-app-data-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-app-data-pv
spec:
  capacity:
    storage: 5Gi         # Must match or exceed PVC request
  accessModes:
    - ReadWriteOnce      # Must match PVC request
  persistentVolumeReclaimPolicy: Retain
  storageClassName: standard-storage # Must match PVC's storageClassName
  local:
    path: /mnt/data/my-app # Path on the node where data will be stored
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - worker-node-01 # The specific node where this local path exists

When using local PVs, the nodeAffinity is crucial. The PV must be bound to a specific node, and the consuming Pod must be scheduled on that same node for the volume to be accessible. Ensure the path exists and has correct permissions on the specified node.

If no matching PV exists, create one using kubectl apply -f my-app-data-pv.yaml.

4. For Dynamic Provisioning (StorageClass with a provisioner)

If your StorageClass specifies a dynamic provisioner, the issue likely lies with that provisioner.

a. Check the provisioner deployment:

First, identify the provisioner name from your StorageClass (kubectl get storageclass <name> -o yaml). Then, check if the corresponding deployment or statefulset for that provisioner is running. Provisioners are often deployed in the kube-system or default namespace.

# Example for NFS-subdir-external-provisioner
kubectl get deployment -n default | grep nfs-client # Or specific namespace for your provisioner
kubectl get pods -n default -l app=nfs-client-provisioner # Use appropriate labels

# Example for local-path-provisioner
kubectl get deployment -n local-path-system # Or specific namespace
kubectl get pods -n local-path-system -l app=local-path-provisioner

Ensure the provisioner pods are in a Running state and healthy. If not, investigate why they failed:

kubectl describe pod <provisioner-pod-name> -n <provisioner-namespace>
kubectl logs <provisioner-pod-name> -n <provisioner-namespace>

b. Common Provisioner Specific Issues:

• NFS-client Provisioner (nfs-subdir-external-provisioner):
  • Configuration: Check the provisioner's deployment/statefulset YAML for correct NFS server IP and path.

    kubectl get deployment nfs-client-provisioner -o yaml -n default # Or your namespace
    

    Look for arguments like -provisionerName=cluster.local/nfs-client, -nfs-server=192.168.1.100, and -nfs-path=/exports/k8s.
  • NFS Server Connectivity: From a Kubernetes node, try to mount the NFS share manually to test connectivity and permissions.

    # On a Kubernetes worker node (via SSH)
    sudo apt update && sudo apt install -y nfs-common
    sudo mkdir -p /mnt/nfs_test
    sudo mount -t nfs 192.168.1.100:/exports/k8s /mnt/nfs_test
    ls -l /mnt/nfs_test
    sudo umount /mnt/nfs_test
    sudo rmdir /mnt/nfs_test
    

    Ensure the NFS share has appropriate export options (e.g., rw,sync,no_subtree_check,no_root_squash or all_squash,anonuid=1000,anongid=1000 depending on your security model) configured on the NFS server. Insufficient permissions will cause provisioning failures.

• Local Path Provisioner (rancher.io/local-path):
  • Host Path: This provisioner uses a local directory on the worker nodes. Ensure the base path (e.g., /var/lib/rancher/k3s/storage for K3s, or /opt/local-path-provisioner) is available and has sufficient disk space on the nodes.
  • Node Selection: If the volumeBindingMode is WaitForFirstConsumer, ensure the Pod can schedule on a node with available local storage.
• Cloud Provider CSI Drivers (e.g., AWS EBS, GCP PD, Azure Disk):
  • IAM/RBAC: Ensure the CSI driver's service account has the necessary IAM roles/permissions to create, attach, and detach volumes in your cloud provider. This is a very common cause of failures.
  • Cloud Quotas: Check if you've hit any volume creation limits or storage quotas in your cloud account.
  • Region/Zone: Ensure your cluster nodes and the requested storage are in compatible regions/zones (especially for ReadWriteOnce volumes tied to a specific zone).
  • kubelet configuration: Ensure kubelet is configured to use the CSI driver correctly (e.g., enable-controller-manager is true, --cloud-provider=external for newer versions).

5. Check System Logs on Nodes

For deep-seated issues, especially with local storage or CSI drivers, inspecting logs on the Kubernetes nodes can be invaluable.

# SSH into a worker node where the provisioner pod *should* be running
journalctl -u kubelet -f # Follow kubelet logs for storage-related events
journalctl -u containerd -f # If using containerd, for container runtime issues
# Or for Docker:
journalctl -u docker -f

Look for errors related to volume mounting, ioctl failures, or issues communicating with storage services.

6. Verify Access Modes and Capacity

Double-check that the accessModes requested by your PVC can actually be provided by your chosen storage solution.

• ReadWriteOnce (RWO): Can be mounted as read-write by a single node. Most block storage supports this.
• ReadOnlyMany (ROX): Can be mounted as read-only by many nodes. Some file storage.
• ReadWriteMany (RWX): Can be mounted as read-write by many nodes. Typically requires shared file systems like NFS or CephFS.

If you request ReadWriteMany but your StorageClass provisions a cloud block device (which is usually RWO), the PVC will stay Pending. Adjust your PVC or use a different StorageClass/provisioner.

Also, confirm that the requested capacity is reasonable and that the underlying storage system has enough free space.

7. Recreate the PVC (Last Resort)

If you've identified and fixed an underlying configuration issue, sometimes the PVC needs to be recreated for Kubernetes to re-evaluate the binding.

Only do this if you are absolutely sure there is no data associated with the PVC, or if the data is ephemeral/backed up. Deleting a PVC can lead to data loss if its reclaimPolicy on the bound PV is Delete and the PV gets garbage collected before you recreate it.

# Backup the PVC definition (optional but recommended)
kubectl get pvc my-app-data-pvc -n my-app-namespace -o yaml > my-app-data-pvc-backup.yaml

# Delete the PVC
kubectl delete pvc my-app-data-pvc -n my-app-namespace

# Wait a few moments for Kubernetes to clean up
# Recreate the PVC from its YAML file
kubectl apply -f my-app-data-pvc.yaml -n my-app-namespace

By systematically working through these steps, you should be able to pinpoint the exact reason your Kubernetes PVC is stuck in Pending status and get your stateful applications running on Ubuntu 20.04 LTS.