Troubleshooting Kubernetes Persistent Volume Claim (PVC) Stuck in Pending Status on Alpine Linux

Resolve Kubernetes PVCs stuck in pending on Alpine Linux nodes. Covers StorageClass, CSI driver, and configuration issues, providing step-by-step solutions for robust storage.

A Kubernetes Persistent Volume Claim (PVC) stuck in Pending status is a common and often frustrating issue for DevOps engineers and system administrators managing containerized applications. This problem indicates that Kubernetes cannot find or provision a suitable Persistent Volume (PV) to fulfill the storage request made by your application. When this occurs on Alpine Linux worker nodes, it can introduce specific challenges due to Alpine's minimalist design and differences in package management and system initialization compared to more common distributions like Ubuntu or CentOS. This guide provides a comprehensive, highly technical approach to diagnosing and resolving PVCs stuck in Pending on Alpine Linux-based Kubernetes clusters.

Symptom & Error Signature

When a Persistent Volume Claim (PVC) is stuck in Pending, your associated Pods will typically remain in a Pending state as well, unable to start because their required storage is unavailable. You'll observe this symptom through kubectl commands:

1. PVC Status:

   kubectl get pvc -n my-app-namespace
   

   NAME            STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
   my-data-pvc     Pending                                      standard       5m
   

2. Pod Status:

   kubectl get pod -n my-app-namespace
   

   NAME                         READY   STATUS    RESTARTS   AGE
   my-app-deployment-xxxx-yyyy  0/1     Pending   0          5m
   

   The pod's description will often show a warning about not being able to mount the volume:

   kubectl describe pod my-app-deployment-xxxx-yyyy -n my-app-namespace
   

   ...
   Events:
     Type     Reason            Age    From               Message
     ----     ------            ----   ----               -------
     Warning  FailedScheduling  5m     default-scheduler  0/3 nodes are available: 3 persistentvolumeclaims "my-data-pvc" not found.
   

   (Note: The persistentvolumeclaims "my-data-pvc" not found message in the pod events indicates the scheduler couldn't even bind the PVC, let alone mount it. The more common PVC pending message is below.)

3. Detailed PVC Events (Crucial for Diagnosis): The most critical information comes from describing the PVC itself, specifically the Events section:

   kubectl describe pvc my-data-pvc -n my-app-namespace
   

   Name:          my-data-pvc
   Namespace:     my-app-namespace
   StorageClass:  standard
   Status:        Pending
   Volume:
   Labels:        <none>
   Annotations:   volume.beta.kubernetes.io/storage-provisioner: example.com/csi-driver
   Finalizers:    [kubernetes.io/pvc-protection]
   Capacity:      
   Access Modes:  
   VolumeMode:    Filesystem
   Mounted By:    <none>
   Events:
     Type     Reason              Age                From                         Message
     ----     ------              ----               ----                         -------
     Warning  ProvisioningFailed  20s (x3 over 1m)   standard-provisioner         failed to provision volume with StorageClass "standard": rpc error: code = Internal desc = volume creation failed: could not connect to storage backend
     Normal   ExternalProvisioning  20s (x3 over 1m)   persistentvolume-controller  waiting for a volume to be created, either by an external provisioner or manually created by an administrator.
   

   Or, for static provisioning:

   Events:
     Type     Reason                Age   From                         Message
     ----     ------                ----  ----                         -------
     Normal   WaitForFirstConsumer  20s   persistentvolume-controller  waiting for first consumer to be created before binding
     Normal   FailedBinding         10s   persistentvolume-controller  no persistent volumes available for this claim and no storage class is set
   

Root Cause Analysis

A PVC gets stuck in Pending when the Kubernetes control plane cannot find or provision a suitable Persistent Volume (PV) to bind to the PVC. This can stem from several underlying issues, often amplified or nuanced when using Alpine Linux for worker nodes:

1. Missing or Misconfigured StorageClass:

   • No Default StorageClass: If the PVC doesn't specify a storageClassName and no default StorageClass is defined cluster-wide, dynamic provisioning cannot occur.
   • Non-existent StorageClass: The storageClassName specified in the PVC does not exist.
   • Misconfigured StorageClass: The specified StorageClass is pointing to a non-existent or improperly configured CSI (Container Storage Interface) driver, or its parameters (e.g., provisioner, parameters) are incorrect.

2. No Available Persistent Volume (PV) for Static Provisioning:

   • If the PVC is meant to bind to a pre-created (statically provisioned) PV, there might be no PV with matching criteria (e.g., storageClassName, capacity, accessModes, labels, nodeSelector).
   • All suitable PVs might already be bound to other PVCs.

3. CSI Driver Issues:

   • Driver Not Installed or Healthy: The CSI driver (e.g., NFS CSI, iSCSI CSI, Ceph CSI) responsible for dynamic provisioning is not deployed, or its pods (provisioner, attacher, node driver) are not running correctly or are in an unhealthy state (e.g., CrashLoopBackOff, Pending).
   • Underlying Storage Connectivity/Configuration: This is where Alpine Linux characteristics become critical.
     • Missing Host Utilities/Kernel Modules on Alpine Nodes: The CSI node driver or the Kubelet itself requires specific host utilities or kernel modules to mount the volume (e.g., nfs-utils for NFS, open-iscsi for iSCSI, xfsprogs for XFS filesystems). Alpine's minimal install often lacks these by default.
     • Network/Firewall Issues: Alpine worker nodes cannot reach the external storage array or server due to network segmentation, firewall rules, or DNS resolution problems.
     • Incorrect CSI Driver Configuration: The CSI driver itself has incorrect credentials, IP addresses, paths, or other parameters needed to interact with the external storage system.

4. Access Mode Mismatch: The PVC requests accessModes (e.g., ReadWriteOnce, ReadWriteMany, ReadOnlyMany) that are incompatible with what the available PVs offer or what the StorageClass's provisioner can support for the underlying storage.

5. Insufficient Capacity: There are no available PVs, or the StorageClass provisioner cannot allocate enough storage to meet the PVC's requests.storage specification.

6. RBAC Permissions: While less common for the Pending status itself (more for failing provisioners), the CSI provisioner's ServiceAccount might lack the necessary Kubernetes RBAC permissions to create PersistentVolume objects or interact with the storage API.

Step-by-Step Resolution

This section outlines a methodical approach to troubleshoot and resolve PVCs stuck in Pending status, with specific considerations for Alpine Linux worker nodes.

1. Inspect the PVC and Related Events

Start by gathering detailed information about the problematic PVC and any associated Pods. The Events section of the PVC description is your primary diagnostic tool.

1. Get PVC Details:

   kubectl describe pvc my-data-pvc -n my-app-namespace
   

   Carefully examine the Status, Volume, StorageClass, and especially the Events section.

   • If Events show FailedBinding or no persistent volumes available for this claim, it points to an issue with static provisioning or a missing StorageClass.
   • If Events show ProvisioningFailed (e.g., failed to provision volume with StorageClass "standard": rpc error: code = Internal desc = volume creation failed), it indicates a problem with dynamic provisioning via the CSI driver. The error message here is crucial.

2. Check Related Pods (if any):

   kubectl describe pod my-app-deployment-xxxx-yyyy -n my-app-namespace
   

   Look for scheduling errors related to the PVC.

2. Verify StorageClass Configuration

If your cluster relies on dynamic provisioning (which is almost always recommended), ensure your StorageClass is correctly defined and operational.

1. Check PVC's StorageClass: Note the StorageClass name from kubectl describe pvc my-data-pvc. If it's empty, and no default StorageClass is defined, this is a major clue.

2. List All StorageClasses:

   kubectl get sc
   

   NAME              PROVISIONER               RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   standard (default)  example.com/csi-driver      Delete          Immediate              true                   2d
   fast-ssd          another.com/fast-csi      Retain          WaitForFirstConsumer   false                  1w
   

   Ensure the StorageClass referenced by your PVC exists. If your PVC did not specify a storageClassName, verify that a default StorageClass (marked with (default)) exists and is functional.

3. Inspect StorageClass Details:

   kubectl describe sc standard
   

   Name:            standard
   IsDefaultClass:  Yes
   Annotations:     storageclass.kubernetes.io/is-default-class=true
   Provisioner:     example.com/csi-driver
   Parameters:      <none>
   MountOptions:    <none>
   AllowVolumeExpansion:  True
   ReclaimPolicy:   Delete
   VolumeBindingMode:   Immediate
   Events:          <none>
   

   Verify the Provisioner field matches the actual CSI driver deployed in your cluster. If the Provisioner is incorrect or the Parameters are misconfigured for your storage backend, the PVC will remain pending.

   If you don't have a StorageClass or if the Provisioner is unknown, you need to deploy a CSI driver for your chosen storage backend (e.g., NFS CSI, iSCSI CSI, local-path provisioner, cloud provider CSI). Refer to your storage vendor's documentation for Kubernetes CSI deployment.

3. Troubleshoot CSI Driver (Dynamic Provisioning)

If a StorageClass is correctly defined but PVCs are still pending with ProvisioningFailed errors, the issue likely lies within the CSI driver.

1. Identify CSI Driver Pods: Typically, CSI driver components run as Pods, often in the kube-system namespace or a dedicated namespace (e.g., csi-nfs-driver).

   kubectl get pod -n kube-system -l app=csi-nfs-controller # Example for NFS CSI
   kubectl get pod -n csi-driver-namespace
   

   Look for pods with names like csi-provisioner, csi-attacher, csi-node, or specific driver names. Ensure all relevant pods are Running and READY.

2. Examine CSI Driver Pod Logs: The logs of the CSI provisioner pod are crucial for understanding why volume creation failed.

   kubectl logs csi-nfs-controller-provisioner-xxxx -n csi-driver-namespace
   kubectl describe pod csi-nfs-controller-provisioner-xxxx -n csi-driver-namespace
   

   Look for errors related to API calls, network connectivity to the storage backend, authentication failures, or capacity issues.

3. Alpine Node Specific Checks for CSI Connectivity: If the CSI driver logs indicate issues connecting to the storage backend or mounting volumes, you'll need to investigate the Alpine worker nodes directly.

   Making direct changes to worker nodes, especially installing packages, can be ephemeral in dynamically scaled clusters or overwritten during node upgrades. For production, consider using custom Alpine images that include necessary tools or ensuring your CSI driver pods include these tools.

   • SSH to an Alpine Worker Node: Identify a worker node where the problematic pod might schedule or where the CSI node driver is running.

     ssh root@<alpine-worker-node-ip>
     

   • Check for NFS Utilities (if using NFS CSI): Alpine uses apk for package management.

     apk info nfs-utils
     

     If not installed, install it:

     apk add nfs-utils
     

     Test connectivity to your NFS server:

     showmount -e <nfs-server-ip>
     # Attempt a manual mount to confirm
     mkdir -p /mnt/test-nfs
     mount -t nfs <nfs-server-ip>:/<path/to/share> /mnt/test-nfs
     df -h /mnt/test-nfs
     umount /mnt/test-nfs
     

     Firewall on Alpine: Alpine uses iptables or nftables. Ensure no rules are blocking NFS ports (2049, 111).

     iptables -L -n | grep -E "2049|111" # Basic check
     

   • Check for iSCSI Utilities (if using iSCSI CSI):

     apk info open-iscsi
     

     If not installed, install it:

     apk add open-iscsi
     

     Ensure the iSCSI daemon is running and enabled on boot:

     rc-service iscsid status
     rc-service iscsid start
     rc-update add iscsid default # Add to default runlevel for persistence
     

     Check iSCSI initiator name and discover targets:

     cat /etc/iscsi/initiatorname.iscsi
     iscsiadm -m discovery -t st -p <iscsi-target-ip>
     

     Firewall on Alpine: Check for blocks on iSCSI port 3260.

   • Other CSI Drivers: Consult the specific CSI driver documentation for any host-level requirements on Alpine Linux and adapt the apk commands accordingly. Missing system dependencies are a very common cause on Alpine.

4. Review Persistent Volume (PV) Availability (Static Provisioning)

If you are using static provisioning (i.e., you manually create PVs) or if dynamic provisioning is failing, check for available PVs.

1. List All PVs:

   kubectl get pv
   

   NAME           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                     STORAGECLASS   REASON   AGE
   my-manual-pv   10Gi       RWO            Retain           Available                                          2h
   bound-pv-1     5Gi        RWO            Delete           Bound       my-app-namespace/other-pvc  standard       1d
   

2. Describe PVs to find a match:

   kubectl describe pv my-manual-pv
   

   Look for an Available PV that matches the PVC's requirements:

   • Capacity: Must be greater than or equal to the PVC's requested storage.
   • Access Modes: Must be compatible (e.g., RWO for ReadWriteOnce, RWX for ReadWriteMany).
   • storageClassName: If the PVC specifies one, the PV must either have the same storageClassName or both PVC and PV omit it for a default binding.
   • Labels/NodeSelector: If the PVC has specific selector or nodeSelector requirements (less common for basic PVCs), the PV must satisfy them.

3. Create or Adjust PV: If no suitable PV is Available, you may need to:

   • Create a new PV that matches the PVC's criteria.
   • Adjust an existing Available PV's configuration to match.

   Example of a static NFS PV manifest:

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: my-nfs-pv
   spec:
     capacity:
       storage: 10Gi
     accessModes:
       - ReadWriteMany # Or ReadWriteOnce, ReadOnlyMany
     persistentVolumeReclaimPolicy: Retain
     storageClassName: "" # Important: set to empty string or remove if PVC also omits storageClassName
     nfs:
       path: /exports/my-data
       server: 192.168.1.100
     mountOptions:
       - hard
       - nfsvers=4.1
   

5. Check Access Mode Compatibility

Ensure the accessModes requested by your PVC are supported by the underlying storage and match the available PVs or what the CSI driver can provision.

1. Review PVC Access Modes:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: my-data-pvc
   spec:
     accessModes:
       - ReadWriteOnce # Could be ReadWriteMany, ReadOnlyMany
     resources:
       requests:
         storage: 10Gi
     storageClassName: standard
   

   If your application requires multiple pods on different nodes to write to the same volume, you must use ReadWriteMany. If you're using ReadWriteOnce but the underlying storage only offers ReadWriteMany, it generally won't bind. The reverse (requesting ReadWriteMany when only ReadWriteOnce is available) will definitely prevent binding.

   Not all storage backends support all access modes. For instance, hostPath PVs are typically ReadWriteOnce (limited to the node they are on), while many distributed file systems (NFS, CephFS) support ReadWriteMany. Block storage (iSCSI, cloud disks) typically only supports ReadWriteOnce for a single attachment.

6. Examine for RBAC Permissions Issues

If CSI driver logs show Permission Denied or similar errors related to Kubernetes API access, check the RBAC setup for the CSI driver's ServiceAccount.

1. Identify CSI Provisioner ServiceAccount: Look at the CSI provisioner pod's YAML to determine its serviceAccountName.

   kubectl get pod csi-nfs-controller-provisioner-xxxx -n csi-driver-namespace -o yaml | grep serviceAccountName
   

2. Inspect Roles/ClusterRoles and Bindings: Check the Role or ClusterRole and corresponding RoleBinding or ClusterRoleBinding for that ServiceAccount. Ensure it has permissions to get, list, watch, create, update, patch, delete persistentvolumes and storageclasses, and potentially other resources depending on the driver.

   kubectl get clusterrole # List cluster roles
   kubectl get clusterrolebinding # List cluster role bindings
   

   This setup is usually part of the CSI driver's deployment manifest, so verify it matches the vendor's recommendations.

7. Clean Up and Reapply

After making configuration changes to StorageClasses, PVs, or CSI drivers, it's often necessary to clean up and reapply your PVC and potentially your Pods.

1. Delete the PVC:

   kubectl delete pvc my-data-pvc -n my-app-namespace
   

   Wait until the PVC is fully deleted (kubectl get pvc should no longer list it).

2. Reapply the PVC:

   kubectl apply -f my-pvc.yaml -n my-app-namespace
   

   Monitor its status:

   kubectl get pvc my-data-pvc -n my-app-namespace
   kubectl describe pvc my-data-pvc -n my-app-namespace
   

3. Delete and Recreate Dependent Pods/Deployments: If the PVC was previously referenced by a Pod or Deployment, you will likely need to delete and recreate them to pick up the newly bound volume.

   kubectl delete deploy my-app-deployment -n my-app-namespace
   kubectl apply -f my-app-deployment.yaml -n my-app-namespace
   

8. Advanced: Debugging Node-Specific Mount Failures

Even if a PVC successfully binds to a PV, the pod might still fail to start if the worker node (Alpine in this case) cannot actually mount the volume. This would manifest as the PVC being Bound but the Pod remaining Pending or entering CrashLoopBackOff with mount errors.

1. Identify Pod's Node:

   kubectl get pod my-app-deployment-xxxx-yyyy -n my-app-namespace -o wide
   

   Note the NODE where the pod is scheduled.

2. SSH to the Alpine Worker Node:

   ssh root@<alpine-worker-node-ip>
   

3. Check Kernel Messages:

   dmesg | tail -n 50
   

   Look for errors related to NFS, iSCSI, XFS, ext4, or generic mount failures.

4. Check System Logs (OpenRC): Alpine typically uses OpenRC. Relevant logs might be found in /var/log/messages or accessed via journalctl if configured.

   grep -i "mount" /var/log/messages
   

   Or, if rsyslog is installed:

   rc-service rsyslog status
   

5. Verify Volume Mounts: Check if the volume is listed as mounted on the node by the kubelet or CSI node driver.

   mount | grep <volume-identifier> # e.g., grep /var/lib/kubelet/pods/<pod-uid>/volumes
   

By systematically working through these steps, from PVC inspection to Alpine node-specific checks and CSI driver diagnostics, you can effectively pinpoint and resolve Kubernetes Persistent Volume Claims stuck in Pending status.