GitHub Actions Runner Out of Disk Space on Windows WSL2 Ubuntu – Troubleshooting Guide

Resolve GitHub Actions 'out of disk space' errors on self-hosted WSL2 Ubuntu runners by clearing caches, pruning Docker, and shrinking virtual disks.

GitHub Actions self-hosted runners running within a Windows Subsystem for Linux 2 (WSL2) Ubuntu environment can encounter "out of disk space" errors during builds. This typically manifests as a build failing abruptly with a No space left on device message, particularly during resource-intensive operations like installing dependencies, compiling code, or building Docker images. While GitHub-hosted runners are generally well-provisioned, self-hosted runners, especially those on WSL2, require careful management of their underlying disk resources.

Symptom & Error Signature

When your GitHub Actions workflow fails due to insufficient disk space, you'll observe errors in the job logs similar to these examples. The exact message can vary depending on the tool or command that attempts to write data when the disk is full.

Run npm install
npm ERR! cb() never called!
npm ERR! There was an error when trying to write to /home/runner/.npm/_cacache/content-v2/sha512/d6/...
npm ERR! A complete log of this run can be found in:
npm ERR!     /home/runner/.npm/_logs/2023-01-01T12_34_56_789Z-debug-0.log
npm ERR! ELOOP: too many symbolic links, copyfile '/tmp/npm-1234567/package' -> '/home/runner/.npm/_cacache/content-v2/sha512/d6/...'
npm ERR! errno -28
npm ERR! ENOSPC: No space left on device, write

Or when building Docker images:

Step 5/10 : RUN npm ci --only=production
 ---> Running in a3b4c5d6e7f8
npm ERR! code ENOSPC
npm ERR! syscall open
npm ERR! path /var/lib/docker/overlay2/temp/a3b4c5d6e7f8_tmp/package.json
npm ERR! errno -28
npm ERR! No space left on device

You might also see:

Error: ENOSPC: no space left on device, mkdir '/home/runner/work/my-repo/my-repo/.cache'

fatal: unable to write new index file: No space left on device

Root Cause Analysis

The "out of disk space" error on a GitHub Actions self-hosted runner running within WSL2 is almost always attributed to the ext4.vhdx virtual disk file associated with your WSL2 distribution growing too large. Here's a breakdown of the underlying reasons:

1. WSL2 Disk Management: WSL2 uses a dynamically expanding VHDX virtual disk (ext4.vhdx) to store the Linux filesystem. While it grows automatically as data is written, it does not automatically shrink when files are deleted. Over time, even after deleting large files, the ext4.vhdx can retain its maximum size, consuming significant space on your host Windows drive.
2. Accumulated Build Caches: CI/CD workflows, especially those involving Node.js (npm, yarn), Java (Maven, Gradle), Rust (Cargo), Python (pip), or system packages (APT), generate substantial caches to speed up subsequent builds. These caches (~/.npm, ~/.cache, ~/.m2, /var/cache/apt, ~/.cargo/registry, etc.) can quickly consume gigabytes of space.
3. Docker Layer Accumulation: If your workflows build or pull Docker images, the Docker daemon within WSL2 will store image layers and build cache in /var/lib/docker. Over time, unused or dangling images, containers, and volumes can accumulate, leading to massive disk consumption. Each build step in a Dockerfile creates a new layer, and failed builds can leave behind intermediate layers.
4. Temporary Files: Processes generate temporary files in /tmp or other designated temp directories. If workflows crash or don't clean up properly, these temporary files can persist and consume space.
5. Workflow Artifacts & Logs: While GitHub Actions handles artifacts, locally generated large log files or intermediate build outputs that aren't cleaned up can contribute to disk exhaustion.
6. Underlying Windows Disk Space: Ultimately, the ext4.vhdx file resides on your Windows host drive. If that drive itself is critically low on space, the WSL2 disk cannot expand further.

Step-by-Step Resolution

Addressing this issue requires a multi-pronged approach, focusing on identifying disk consumers, cleaning up accumulated data, and maintaining the WSL2 virtual disk.

1. Assess Current Disk Usage

The first step is to understand what is consuming space within your WSL2 Ubuntu environment.

1. Start your WSL2 Ubuntu instance:

   wsl
   

2. Check overall disk usage:

   df -h
   

   Look for the filesystem mounted at / (usually /dev/sdb or similar). This will show the total size, used space, and available space.

3. Identify top-level directories consuming space:

   sudo du -sh /* | sort -rh | head -n 10
   

   This command lists the sizes of directories directly under the root (/), sorted from largest to smallest, showing the top 10. Pay close attention to /var, /usr, /home, and /opt.

4. Deep-dive into known culprits:

   • Docker:

     sudo du -sh /var/lib/docker
     

   • User Caches (for the runner user or your user):

     sudo du -sh /home/runner/.cache
     sudo du -sh /home/runner/.npm
     sudo du -sh /home/runner/.m2 # If using Maven/Java
     sudo du -sh /home/runner/.cargo # If using Rust
     

   • APT Cache:

     sudo du -sh /var/cache/apt
     

   • Temporary Files:

     sudo du -sh /tmp
     sudo du -sh /var/tmp
     

2. Clean Up Docker Resources

Docker is a frequent culprit for disk space issues. Ensure your self-hosted runner is not actively running a critical build before performing aggressive cleanups.

1. Prune all unused Docker data:

   docker system prune -a
   

   docker system prune -a will remove all stopped containers, all networks not used by at least one container, all dangling images, all build cache, and all dangling volumes. This is a powerful command and should be used with caution on production systems, but it's often necessary for self-hosted runners. It will prompt for confirmation.

2. Alternatively, more granular pruning:

   • Remove stopped containers:

     docker container prune
     

   • Remove dangling images (images not associated with any container):

     docker image prune
     

   • Remove dangling volumes:

     docker volume prune
     

   • Remove build cache:

     docker builder prune
     

3. Clear Build Caches

Various package managers and build tools maintain local caches. Clearing these can free up significant space.

1. APT (Ubuntu package manager):

   sudo apt clean
   sudo apt autoremove --purge
   

   apt clean clears the local repository of retrieved package files. apt autoremove removes packages that were automatically installed to satisfy dependencies for other packages and are no longer needed.

2. NPM (Node.js Package Manager):

   npm cache clean --force
   

   npm cache clean --force will forcefully remove all data from the npm cache. This can significantly slow down subsequent npm install operations until the cache is rebuilt.

3. Yarn (Node.js Package Manager):

   yarn cache clean
   

4. Maven (Java Build Tool): The Maven local repository (~/.m2/repository) can become very large.

   rm -rf ~/.m2/repository/*
   

   This will delete all downloaded artifacts. Maven will re-download them as needed.

5. Cargo (Rust Package Manager):

   • Project-specific cache: Navigate to your project directory and run:

     cargo clean
     

   • Global registry and git caches:

     rm -rf ~/.cargo/registry
     rm -rf ~/.cargo/git
     

     These caches are global and will be re-downloaded on demand.

6. Pip (Python Package Installer):

   pip cache purge
   

4. Manage Temporary Files

Remove accumulated temporary files that were not cleaned up.

sudo rm -rf /tmp/*
sudo rm -rf /var/tmp/*

Ensure no critical processes are currently writing to /tmp before running rm -rf /tmp/*. This is generally safe on an idle runner.

5. Shrink the WSL2 Virtual Disk (ext4.vhdx)

After performing cleanups within your WSL2 distribution, the ext4.vhdx file on your Windows host won't automatically shrink. You need to manually compact it.

1. Shut down all WSL2 distributions: Close all WSL2 terminal windows and then run in a Windows PowerShell or Command Prompt:

   wsl --shutdown
   

2. Locate the ext4.vhdx file: The default location is typically: %LOCALAPPDATA%PackagesCanonicalGroupLimited.UbuntuonWindows_...LocalStateext4.vhdx Replace ... with your specific Ubuntu version package ID. A common path looks like: C:Users<YourUser>AppDataLocalPackagesCanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgscLocalStateext4.vhdx

3. Use diskpart to compact the VHDX: Open Windows PowerShell or Command Prompt as Administrator and run:

   diskpart
   

   Within the DISKPART> prompt:

   select vdisk file="C:Users<YourUser>AppDataLocalPackagesCanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgscLocalStateext4.vhdx"
   # Verify the path is correct before proceeding.
   attach vdisk readonly
   compact vdisk
   detach vdisk
   exit
   

   Ensure you have a backup of your WSL2 distribution or critical data before performing disk operations with diskpart. While compact vdisk is generally safe, always exercise caution.

6. Allocate More Disk Space (If Necessary)

If your host Windows drive is consistently running low, or the default WSL2 configuration is too restrictive:

1. Move WSL2 distribution to another drive: If your C: drive is full, you can move the WSL2 distribution to a drive with more space using wsl --export and wsl --import.

   wsl --export Ubuntu-20.04 D:wslubuntu.tar
   wsl --unregister Ubuntu-20.04 # WARNING: This deletes the original!
   wsl --import Ubuntu-20.04 D:wsldata D:wslubuntu.tar --version 2
   

   This exports your distribution to a .tar file, unregisters (deletes) the original, and then imports it to a new location (D:wsldata).

2. Increase WSL2 limits (memory/CPU, not direct disk): While not directly disk-related, if your builds are memory-intensive and causing other issues, you can configure .wslconfig. Create or edit C:Users<YourUser>.wslconfig:

   [wsl2]
   memory=8GB  # Adjust as needed
   processors=4 # Adjust as needed
   

   Then wsl --shutdown for changes to take effect.

7. Configure GitHub Actions Workflow for Disk Management

To prevent recurrence, integrate cleanup steps directly into your GitHub Actions workflow for self-hosted runners.

name: CI with Disk Management

on: [push, pull_request]

jobs:
  build:
    runs-on: self-hosted # Or your specific self-hosted runner label

    steps:
    - name: Checkout code
      uses: actions/checkout@v4

    - name: Display initial disk usage
      run: df -h

    - name: Clean up Docker before build (if applicable)
      run: |
        echo "Pruning Docker system..."
        docker system prune -a -f || true # -f for force, || true to prevent job failure if nothing to prune
        docker builder prune -f || true
        echo "Docker cleanup complete."

    - name: Clean up npm cache (if applicable)
      run: |
        echo "Cleaning npm cache..."
        npm cache clean --force || true
        echo "npm cache cleanup complete."

    - name: Set up Node.js
      uses: actions/setup-node@v4
      with:
        node-version: '20'

    - name: Install dependencies
      run: npm ci

    - name: Run tests
      run: npm test

    - name: Build project
      run: npm run build

    - name: Display final disk usage
      run: df -h

    - name: Final cleanup after build
      if: always() # Run even if previous steps fail
      run: |
        echo "Performing final cleanup..."
        sudo apt clean || true
        sudo apt autoremove -y || true
        sudo rm -rf /tmp/* || true
        # Consider specific project cleanup here, e.g., removing build artifacts
        echo "Final cleanup complete."

Automating cleanup in your workflow ensures that your self-hosted runner's disk space is managed proactively, reducing the chance of future build failures. The || true suffix on cleanup commands prevents the workflow step from failing if, for instance, npm cache clean finds nothing to clean.

8. Monitor Disk Usage

Regularly monitor the disk usage of your self-hosted runner. You can:

• Add df -h steps at various points in your workflow.
• Implement external monitoring solutions (e.g., Prometheus Node Exporter with Grafana) on the Windows host and within WSL2 to track disk utilization trends.

By systematically applying these steps, you can effectively resolve and prevent "out of disk space" errors for your GitHub Actions self-hosted runners on Windows WSL2 Ubuntu.