GitHub Actions Workflow Failed: Runner Out of Disk Space on CentOS Stream / Rocky Linux

Troubleshoot and resolve 'runner out of disk space' errors in GitHub Actions on CentOS Stream / Rocky Linux builds by optimizing cache, artifacts, and runner storage.

When running GitHub Actions workflows, especially complex builds involving large dependencies or container images, it's not uncommon to encounter frustrating "out of disk space" errors. This issue is particularly prevalent on self-hosted runners where disk provisioning might be overlooked or aggressive caching strategies fill up available storage. This guide provides a highly technical, accurate, and step-by-step approach to diagnosing and resolving disk space exhaustion on GitHub Actions self-hosted runners running CentOS Stream or Rocky Linux.

Symptom & Error Signature

The most common symptom is a GitHub Actions workflow job failing abruptly, often after prolonged activity during a build step (e.g., dependency installation, large compilation, Docker image build). The specific error messages can vary depending on the tool or command that hits the storage limit.

You might see messages similar to these in your workflow logs:

Error: ENOSPC: no space left on device, write

No space left on device

Failed to write to temporary file: No space left on device

... (during a 'npm install', 'composer install', 'pip install', 'mvn clean install', 'go build', 'docker build' step)
...
Error: build failed: failed to create task for ...: no space left on device

If you manage to SSH into your self-hosted runner while a build is failing or immediately after, you'd observe high disk utilization:

[runner_user@my-runner ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G  9.1M  3.9G   1% /run
/dev/sda2        50G   48G  2.0G  96% /        <-- Problem here!
/dev/sda1       976M  250M  660M  28% /boot
tmpfs           784M   12K  784M   1% /run/user/1000

And investigating the runner's work directory (_work) could reveal large directories:

[runner_user@my-runner _work]$ du -sh *
4.0K	actions-runner
8.2G	_actions
15G	my-repo
12G	my-other-repo

Root Cause Analysis

Understanding the underlying reasons for disk space exhaustion on a GitHub Actions runner is crucial for effective resolution.

1. GitHub Actions Runner Workspace (_work directory):

   • Each workflow run checks out the repository and related actions into the runner's _work directory.
   • Over time, multiple repositories and branches, especially large ones, can accumulate significant data.
   • The _work/_actions directory caches downloaded actions, which can also grow large if many different actions are used.

2. Build Artifacts and Dependencies:

   • Node.js: node_modules directories can easily consume gigabytes.
   • Java/Maven/Gradle: ~/.m2/repository or ~/.gradle caches, target/ directories.
   • Python: venv/ directories or large site-packages within environments.
   • PHP/Composer: vendor/ directories.
   • Go: GOCACHE, GOMODCACHE.
   • Rust: target/ directories.
   • Temporary files: Compilers, package managers, and build tools often create temporary files in /tmp or within the build directory that might not be cleaned up automatically.

3. Docker/Podman Images, Layers, and Volumes:

   • If your workflows build or pull Docker/Podman images, these tools maintain a local cache of image layers and containers.
   • Dangling images, unused containers, and unreferenced volumes can quickly consume vast amounts of disk space.
   • Each docker build step creates intermediate layers, which persist until explicitly pruned.

4. Inefficient Caching Strategy:

   • Lack of or improper use of actions/cache leads to re-downloading and rebuilding dependencies on every workflow run, increasing both build time and disk usage.

5. Ephemeral Storage Limitations:

   • While this guide focuses on self-hosted runners, even with seemingly ample disk space, the specific partition where / or /var/lib/docker resides might be too small.

6. Logging:

   • Excessive logging by applications or system services can fill up /var/log, though this is less common as the primary cause for workflow failures directly due to disk space.

Step-by-Step Resolution

Here’s a comprehensive approach to diagnose and resolve disk space issues on your CentOS Stream / Rocky Linux GitHub Actions runner.

1. Initial Diagnosis on the Runner

First, connect to your self-hosted runner via SSH to gather diagnostic information.

1. Check overall disk usage:

   df -h
   

   Identify which mount points are nearing capacity, especially / (root filesystem) and /var/lib/docker (if on a separate partition).

2. Locate the GitHub Actions runner installation directory: Typically, this is /opt/actions-runner or ~/actions-runner. Navigate into it.

3. Analyze the runner's workspace (_work directory): This is where repositories are checked out and builds occur.

   cd /opt/actions-runner/_work # Or wherever your runner is installed
   sudo du -sh *
   

   This command will show you the total size of each directory within _work. The repository directories (e.g., my-repo/) will likely be the largest.

4. Identify large directories within a specific repository workspace: If a repository's directory is large, drill down into it.

   cd my-repo/my-project/ # Replace with actual path
   sudo ncdu -x . # Interactive disk usage analyzer, install with 'sudo dnf install ncdu'
   

   ncdu is incredibly useful for interactively finding large files and directories.

5. Check Docker/Podman storage: If your builds use containers, check the space consumed by images, containers, and volumes.

   sudo docker system df # For Docker
   sudo podman system df # For Podman
   

If df -h shows / (root filesystem) is almost full, it's critical to identify the large directories within it, which might include /var/lib/docker, /opt/actions-runner, /tmp, or even /var/log.

2. Optimize GitHub Actions Workflow Configuration

Applying these changes directly in your .github/workflows/*.yml files can significantly reduce disk pressure during builds.

1. Implement or Improve Caching with actions/cache: This is often the single most effective way to reduce disk usage and build times by caching dependencies.

   name: Build and Test
   on: [push, pull_request]
   
   jobs:
     build:
       runs-on: self-hosted
       steps:
         - uses: actions/checkout@v4
   
         - name: Cache Node.js modules
           uses: actions/cache@v4
           with:
             path: ~/.npm # Or 'node_modules' if you prefer local cache
             key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
             restore-keys: |
               ${{ runner.os }}-node-
   
         - name: Set up Node.js
           uses: actions/setup-node@v4
           with:
             node-version: '18'
   
         - name: Install dependencies
           run: npm ci
   
         - name: Cache Maven dependencies
           uses: actions/cache@v4
           with:
             path: ~/.m2/repository
             key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
             restore-keys: |
               ${{ runner.os }}-maven-
   
         - name: Set up Java
           uses: actions/setup-java@v4
           with:
             distribution: 'temurin'
             java-version: '17'
   
         - name: Build with Maven
           run: mvn clean install -DskipTests
   
         # Add more cache steps for other languages/tools (e.g., Composer, pip, Go modules)
   

   Ensure key changes only when dependencies change, and restore-keys provides a fallback for partial matches. Cache path should point to the actual directory where dependencies are stored (e.g., node_modules, ~/.npm, ~/.m2/repository, ~/.cache/pip).

2. Manage Build Artifacts: Only upload artifacts that are absolutely necessary for subsequent steps or release. Avoid uploading entire node_modules or target directories.

   - name: Upload only essential build artifacts
     uses: actions/upload-artifact@v4
     with:
       name: my-app-build
       path: dist/ # Or target/my-app.jar, etc.
       # Don't include:
       # - node_modules/
       # - vendor/
       # - .git/
   

3. Minimize Docker Image Layers and Size: If using Docker in your workflow, optimize your Dockerfiles.

   • Use multi-stage builds.
   • Use slimmer base images (e.g., alpine variants).
   • Combine RUN commands to reduce layers.
   • Remove build dependencies after they are no longer needed.

   # Dockerfile Example with Multi-stage build
   # Stage 1: Build
   FROM node:18-alpine AS build-stage
   WORKDIR /app
   COPY package*.json ./
   RUN npm ci --only=production
   COPY . .
   RUN npm run build
   
   # Stage 2: Run
   FROM nginx:stable-alpine AS production-stage
   COPY --from=build-stage /app/dist /usr/share/nginx/html
   EXPOSE 80
   CMD ["nginx", "-g", "daemon off;"]
   

4. Clean up Runner Workspace Post-Job/Post-Workflow: By default, actions/checkout does not clean up the workspace. You can configure it to do so.

   - name: Checkout repository and clean existing workspace
     uses: actions/checkout@v4
     with:
       clean: true # Removes previous checkouts and untracked files
   

   For more aggressive cleanup between jobs or at the end of a workflow, you might add a step to remove large directories:

   - name: Clean up specific build directories
     if: always() # Run even if previous steps failed
     run: |
       rm -rf node_modules/
       rm -rf vendor/
       rm -rf target/
       # Be cautious with 'rm -rf /tmp/*' - ensure no critical temp files are needed by other processes.
     working-directory: ${{ github.workspace }}/your-project-path
   

3. Self-Hosted Runner Specific Optimizations (CentOS Stream / Rocky Linux)

These steps involve managing the runner server itself.

1. Increase Disk Capacity: The most straightforward solution if you consistently hit limits is to provision more disk space for your runner VM/server. Ensure the partition housing / and especially /opt/actions-runner and /var/lib/docker has sufficient capacity (e.g., 100GB+ for active CI/CD).

   If you need to resize disks, ensure you back up critical data first and follow your cloud provider's or hypervisor's specific instructions for disk resizing and filesystem expansion (e.g., growpart and xfs_growfs for XFS or resize2fs for EXT4).

2. Periodically Clean Up Docker/Podman Resources: These commands are crucial for freeing up space used by container images, layers, and volumes. Schedule them to run regularly.

   # For Docker: Remove all stopped containers, dangling images, and unused networks/volumes
   sudo docker system prune -a --volumes --force
   
   # For Podman: Remove all stopped containers, dangling images, and unused networks/volumes
   sudo podman system prune -a --volumes --force
   

   docker system prune -a --volumes (or podman system prune) will remove ALL unused containers, networks, images, and volumes. This is powerful but can be destructive if you have local data in volumes that are not managed by your CI/CD. Use with caution. Consider docker image prune or docker volume prune for more granular control.

3. Configure Docker to Limit Log Size: Container logs can consume significant disk space over time. Configure log-opts for the Docker daemon.

   Create or edit /etc/docker/daemon.json:

   {
     "log-driver": "json-file",
     "log-opts": {
       "max-size": "10m",
       "max-file": "3"
     }
   }
   

   This limits container logs to 10MB per file, with a maximum of 3 files (total 30MB per container). Restart Docker after making changes:

   sudo systemctl restart docker
   

4. Automate Runner Workspace Cleanup (Cron Job): While actions/checkout with clean: true helps, a cron job can clean up old, inactive runner workspaces or artifacts that might have been left behind.

   Edit the crontab for the user running the GitHub Actions runner (or root if the runner is run as root, though not recommended).

   crontab -e
   

   Add a line to clean up old files in the _work directory. This example removes files older than 7 days. Adjust the path and age as needed.

   # Clean up GitHub Actions runner workspace directories older than 7 days
   0 3 * * * find /opt/actions-runner/_work/ -maxdepth 2 -type d -name "*" -mtime +7 -exec rm -rf {} +
   

   Be extremely careful with rm -rf commands. Test the find command without -exec rm -rf {} + first (e.g., find ... -print) to ensure it targets only what you intend to delete.

5. Clean up System Temporary Files: CentOS Stream / Rocky Linux uses systemd-tmpfiles for managing temporary files. By default, it cleans up /tmp and /var/tmp. You can verify its configuration.

   grep -r 'tmp' /usr/lib/tmpfiles.d/ /etc/tmpfiles.d/
   

   If you have specific applications that leave large temporary files elsewhere, you can define custom tmpfiles.d configurations.

   To manually clean up /tmp (be careful if other processes are active):

   sudo rm -rf /tmp/*
   

6. Use tmpfs for Ephemeral Build Data: For very large, temporary build artifacts that are only needed during a single job run and don't require persistence (e.g., intermediate compiler outputs), consider using tmpfs (RAM filesystem).

   You can mount a tmpfs into a specific build directory at the start of a job and unmount it at the end. This is more advanced and requires careful workflow scripting.

   Example in a workflow step:

   - name: Create tempfs for build
     run: |
       sudo mkdir -p /mnt/build-tmp
       sudo mount -t tmpfs -o size=10G tmpfs /mnt/build-tmp
       # Change permissions if runner user isn't root
       sudo chown ${{ github.runner_name }}: /mnt/build-tmp
     # Ensure this runs as early as possible
   
   - name: Perform build in tempfs
     working-directory: /mnt/build-tmp
     run: |
       # Copy necessary source files here
       cp -R ${{ github.workspace }}/my-repo/* .
       # Perform your build (e.g., mvn clean install)
       # Your target/ node_modules etc. will be in /mnt/build-tmp
     # Your build tool might need to be configured to output here
   
   - name: Clean up tempfs
     if: always() # Crucial to unmount even on failure
     run: |
       sudo umount /mnt/build-tmp
       sudo rmdir /mnt/build-tmp
   

7. Monitor Disk Usage: Implement monitoring for your self-hosted runner's disk usage (e.g., Prometheus Node Exporter, Nagios, Zabbix). Set alerts when disk utilization exceeds a threshold (e.g., 80-90%) to proactively address issues before they cause workflow failures.

By systematically applying these strategies, you can effectively manage disk space on your CentOS Stream / Rocky Linux GitHub Actions runners, ensuring reliable and efficient CI/CD workflows.