Troubleshooting: GitHub Actions ‘Runner Out of Disk Space’ on Debian 12 Bookworm

Resolve GitHub Actions 'out of disk space' errors on Debian 12 Bookworm self-hosted runners. Optimize CI/CD builds, clean caches, and manage Docker resources for seamless deployments.

Introduction

As an experienced Systems Administrator and DevOps engineer, encountering "out of disk space" errors during CI/CD builds is a classic challenge, particularly with self-hosted GitHub Actions runners. When your GitHub Actions workflow fails with messages indicating insufficient disk space on a Debian 12 Bookworm runner, it typically means your build process is consuming more storage than the runner's underlying filesystem can provide. This guide provides a highly technical, accurate, and actionable approach to diagnose and resolve this issue, ensuring your build pipelines remain robust and efficient.

This problem commonly manifests during stages involving large dependency installations (e.g., node_modules, vendor, venv), Docker image builds, or extensive temporary file generation. While GitHub-hosted runners have generous, albeit fixed, disk allocations, self-hosted runners require meticulous management to prevent such failures.

Symptom & Error Signature

The primary symptom is a failed GitHub Actions workflow run. Upon inspecting the job logs, you'll observe errors indicating disk saturation. The exact error messages can vary depending on the tool or command attempting to write to disk, but they generally point to the same underlying issue.

Here are common error signatures you might encounter:

Error: No space left on device

fatal: Out of disk space

Failed to write to disk. Disk quota exceeded

df: /var/lib/docker/overlay2: No such file or directory
df: no file systems processed
Error: ENOENT: no such file or directory, stat '/github/workspace/node_modules'

Sometimes, the error might appear indirectly as a build tool failing without an explicit "disk space" message, but preceding logs might show high disk usage or warnings. For Docker-related builds, you might see failures during image layers extraction or creation.

Run npm install
...
npm ERR! code ENOSPC
npm ERR! syscall write
npm ERR! path /github/workspace/node_modules/some-package/index.js
npm ERR! errno -28
npm ERR! ENOSPC: no space left on device, write

Error response from daemon: driver failed programming external connectivity on endpoint my-container (uuid): Error starting userland proxy: listen tcp4 0.0.0.0:80: bind: address already in use

While the above Docker error doesn't explicitly state "disk space", it can be a symptom if temporary Docker volumes or build caches are consuming too much inode space or disk, preventing further daemon operations.

Root Cause Analysis

The "runner out of disk space" error on a Debian 12 Bookworm self-hosted runner stems from one or a combination of the following underlying issues:

1. Cumulative Build Artifacts and Caches: Over time, if not properly managed, self-hosted runners accumulate large build caches (e.g., actions/cache entries, Maven local repositories, npm/yarn caches), Docker build caches (/var/lib/docker), and temporary files from previous runs.
2. Inefficient Docker Layer Management: When building Docker images, especially without multi-stage builds or proper .dockerignore files, intermediate layers and build contexts can become excessively large. If docker system prune is not regularly executed, old images, containers, volumes, and build caches consume significant space.
3. Large Project Dependencies: Modern applications often have vast dependency trees (e.g., node_modules for JavaScript, vendor for Go/PHP, .venv for Python). These can quickly swell, especially if multiple projects or branches are built on the same runner without proper cleanup.
4. Temporary Files and Logs: Some build processes generate large temporary files or extensive log outputs that are not cleared after the job completes, leading to gradual disk consumption.
5. Insufficient Initial Disk Provisioning: The self-hosted runner's virtual machine or host machine might have been provisioned with inadequate disk space from the outset, unable to handle the peak storage requirements of complex builds.
6. Runner Not Ephemeral: If the self-hosted runner is a persistent instance that runs multiple jobs sequentially, it accumulates state and temporary files from each job. Without a mechanism to reset its state, disk space will inevitably deplete.

Step-by-Step Resolution

Addressing this issue requires a multi-pronged approach, combining workflow optimization, runner maintenance, and potentially infrastructure adjustments.

1. Analyze Disk Usage on the Runner

Before making changes, identify what is consuming space. If you have SSH access to your Debian 12 self-hosted runner, log in and use standard Linux tools:

# Check overall disk usage
df -h

# Check inode usage (if many small files are the issue)
df -i

# Recursively check directory sizes (start from problematic directories like /home/runner/work or /var/lib/docker)
sudo du -sh /home/runner/work/*
sudo du -sh /var/lib/docker/*
sudo du -sh /var/lib/apt/lists

Focus on /home/runner/work/, where GitHub Actions workflows execute, and /var/lib/docker if Docker is involved in your builds. These are common culprits for self-hosted runner disk space issues.

2. Optimize Workflow Temporary File Cleanup

Ensure your workflow explicitly cleans up large, temporary files or directories after they are no longer needed.

jobs:
  build:
    runs-on: self-hosted
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Install dependencies (e.g., Node.js)
        run: |
          npm install

      - name: Run build process
        run: |
          npm run build

      - name: Archive production artifacts
        uses: actions/upload-artifact@v4
        with:
          name: my-app-build
          path: dist/

      - name: Clean up node_modules and other temp files
        if: always() # Ensure this step runs even if previous steps fail
        run: |
          echo "Cleaning up build artifacts and caches..."
          sudo rm -rf node_modules
          sudo rm -rf .cache
          sudo apt-get clean # Clean up apt cache on the runner itself

Use rm -rf with extreme caution. Always specify the exact directory to remove to avoid accidental data loss. Using if: always() for cleanup steps is a good practice to ensure they run regardless of previous step outcomes.

3. Efficient Docker Image & Container Management

If your workflow builds or uses Docker images, this is often a major source of disk consumption.

a. Use Multi-Stage Builds

Refactor your Dockerfile to use multi-stage builds. This allows you to discard build-time dependencies and intermediate layers, resulting in smaller final images.

# Stage 1: Build
FROM node:20-slim AS builder
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build

# Stage 2: Run
FROM nginx:alpine
COPY --from=builder /app/dist /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

b. Utilize .dockerignore

Prevent unnecessary files from being added to your Docker build context. This reduces the size of layers and speeds up builds.

# .dockerignore example
node_modules
.git
.vscode
npm-debug.log
dist
tmp/

c. Prune Docker System on Self-Hosted Runners

For persistent self-hosted runners, regularly prune Docker resources. This removes stopped containers, dangling images, unused networks, and build cache.

jobs:
  build:
    runs-on: self-hosted
    steps:
      # ... your build steps involving Docker ...

      - name: Prune Docker System
        if: always() # Important for cleanup
        run: |
          echo "Pruning Docker system..."
          docker system prune -a --volumes -f
          # For buildx (BuildKit) caches specifically
          docker buildx prune -f

docker system prune -a --volumes -f is an aggressive command. It removes all stopped containers, all unused networks, all dangling images, all build cache, and all unused volumes. Use it only on self-hosted runners dedicated to CI/CD where state persistence between jobs is not desired.

4. Fine-Tune actions/cache Usage

The actions/cache action is powerful but can also consume significant disk space if not managed carefully.

a. Scope Caches Appropriately

Ensure your cache keys are granular. Avoid caching entire directories that contain frequently changing files or temporary data.

# Good: Cache node_modules based on package-lock.json
- name: Cache Node.js modules
  uses: actions/cache@v4
  with:
    path: ~/.npm
    key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-node-

# Potentially problematic (too broad, might cache too much)
# - name: Cache entire workspace (avoid this unless necessary)
#   uses: actions/cache@v4
#   with:
#     path: .
#     key: ${{ runner.os }}-project-${{ github.sha }}

b. Consider Cache Eviction Policies

While GitHub-hosted runners have automatic cache eviction, self-hosted runners technically store caches directly on their filesystem. If you have many different cache keys, they can accumulate. You might need to manually delete specific cache entries via the GitHub UI (Repository Settings > Actions > Caches).

c. Use --mount=type=cache with Docker BuildKit

For Docker builds, consider using BuildKit's cache mounts for dependencies like npm or apt caches, which are more efficient than actions/cache for Docker layers.

# Example Dockerfile with BuildKit cache mount
FROM node:20-slim AS builder
WORKDIR /app
COPY package*.json ./
# Cache npm downloads using BuildKit mount
RUN --mount=type=cache,target=/root/.npm 
    npm install

This requires your runner's Docker daemon to be running with BuildKit, which is usually the default for newer Docker installations.

5. Consider Ephemeral Runners

The most robust solution for self-hosted runner disk space issues is to treat runners as ephemeral. This means each job gets a fresh, clean runner instance, discarding all state and disk usage from previous jobs.

a. Orchestration for Ephemeral Runners

Use an orchestration system like Kubernetes, HashiCorp Nomad, or cloud-specific autoscaling groups (e.g., AWS EC2 Auto Scaling, Azure VM Scale Sets) to spin up a new runner instance for each job, and tear it down immediately after the job completes.

Example for a systemd managed runner that exits after one job:

Modify the svc.sh script or the systemd service file (/etc/systemd/system/actions-runner.service) to add the --once flag.

# Example /etc/systemd/system/actions-runner.service (simplified)
[Unit]
Description=GitHub Actions Runner
After=network.target

[Service]
ExecStart=/home/runner/actions-runner/run.sh --once
WorkingDirectory=/home/runner/actions-runner
User=runner # Or your dedicated runner user
Group=runner # Or your dedicated runner group
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

You would then need an external mechanism (e.g., a cron job, a cloud function, or a custom controller) to register new runners with GitHub when needed.

Implementing ephemeral runners requires advanced orchestration knowledge. Ensure your registration and de-registration logic is robust to avoid orphaned runners or authentication issues.

6. Increase Runner Disk Capacity (Last Resort)

If all optimization efforts still result in disk space issues, the runner's underlying storage might be genuinely insufficient for your build processes. This is often the case for very large projects or those requiring extensive Docker image layering.

a. Resize Virtual Machine Disk

If your self-hosted runner is a virtual machine (e.g., on AWS EC2, Azure VM, Google Cloud, VMware, Proxmox), expand its virtual disk.

# Assuming you've expanded the underlying VM disk, you then need to expand the filesystem.
# First, identify the disk and partition. This might be /dev/sda or /dev/vda.
sudo fdisk -l

# Re-scan partitions if your cloud provider automatically expanded it.
# This varies by cloud/hypervisor.
# For some cloud images (e.g., cloud-init), this might happen automatically on reboot.
# Otherwise, you might need to use `growpart` or `parted`.
# Example using growpart (if installed, apt install cloud-guest-utils):
sudo growpart /dev/vda 1 # Expand the first partition on /dev/vda

# Then, expand the filesystem (ext4 example)
sudo resize2fs /dev/vda1 # Adjust /dev/vda1 to your actual root partition

# Verify the new size
df -h

Always back up your runner's disk before attempting resize operations. Incorrect partition or filesystem resizing can lead to data loss and an unbootable system. Consult your cloud provider's documentation for their specific disk resizing procedures.

b. Attach Additional Storage

Alternatively, you could attach a separate block storage device (e.g., EBS volume, Azure Disk) and mount it to a directory like /var/lib/docker to offload Docker storage.

# Example: Attach and format a new disk (/dev/sdb) and mount to /var/lib/docker
# CAUTION: This will move your existing /var/lib/docker data, back up first!

# 1. Stop Docker
sudo systemctl stop docker

# 2. Back up existing Docker data (crucial!)
sudo mv /var/lib/docker /var/lib/docker_backup

# 3. Format the new disk (assuming /dev/sdb and you want ext4)
sudo mkfs.ext4 /dev/sdb

# 4. Create mount point
sudo mkdir -p /var/lib/docker

# 5. Mount the new disk
sudo mount /dev/sdb /var/lib/docker

# 6. Restore Docker data
sudo mv /var/lib/docker_backup/* /var/lib/docker/

# 7. Add to /etc/fstab for persistent mount
echo '/dev/sdb /var/lib/docker ext4 defaults 0 0' | sudo tee -a /etc/fstab

# 8. Start Docker
sudo systemctl start docker

# 9. Verify
df -h /var/lib/docker

By systematically applying these solutions, you can effectively resolve "out of disk space" errors on your Debian 12 GitHub Actions self-hosted runners and maintain a robust CI/CD pipeline.