WSL2 Ubuntu: Troubleshooting & Fixing ‘No Space Left on Device’ Due to Inode Exhaustion

Resolve 'No Space Left on Device' errors in WSL2 Ubuntu when df -h shows free space but df -i reports 100% inode usage. Learn to identify and clear inode hogs.

Resolve 'No Space Left on Device' errors in WSL2 Ubuntu when df -h shows free space but df -i reports 100% inode usage. Learn to identify and clear inode hogs.

Introduction

You're working within your Windows Subsystem for Linux 2 (WSL2) Ubuntu environment, and suddenly your applications start failing with "No space left on device" or similar errors. You instinctively run df -h to check disk usage, only to find that there appears to be ample free space. Confused, you might then run df -i, revealing the true culprit: 100% inode utilization. This guide will walk you through understanding why this happens in WSL2 and provide a comprehensive, step-by-step resolution.

Symptom & Error Signature

The primary symptom is applications failing to create new files or directories, or even write to existing ones, reporting disk space-related errors, despite df -h indicating available capacity.

Typical output when running df -h within your WSL2 instance:

user@hostname:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb        251G   50G  201G  20% /
tmpfs           7.8G     0  7.8G   0% /mnt/wsl
none            7.8G  4.0K  7.8G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            7.8G     0  7.8G   0% /run/shm
none            7.8G     0  7.8G   0% /run/user

Notice the Use% for /dev/sdb (your WSL2 root filesystem) is only 20%, showing plenty of free space (201G).

However, checking inode usage (df -i) reveals a different story:

user@hostname:~$ df -i
Filesystem        Inodes   IUsed   IFree IUse% Mounted on
/dev/sdb       163840000 163840000       0  100% /
tmpfs            2048035       1 2048034    1% /mnt/wsl
none             2048035       2 2048033    1% /run
none             2048035       3 2048032    1% /run/lock
none             2048035       4 2048031    1% /run/shm
none             2048035       5 2048030    1% /run/user

Here, IUse% for /dev/sdb is 100%, indicating that all available inodes have been consumed.

Common application errors you might encounter:

No space left on device
Cannot create file: Disk quota exceeded
Error creating directory: Input/output error

Root Cause Analysis

Inodes (index nodes) are a fundamental data structure in Unix-like file systems (like ext4 used by WSL2) that describe a file system object such as a file or a directory. Each file and directory on a Linux system requires exactly one inode. An inode stores metadata about the file, such as its size, owner, permissions, timestamps, and a pointer to its data blocks on the disk. It does not store the file's name or its actual content.

The paradox of df -h showing free space while df -i shows 100% usage occurs when your system has created an extremely large number of very small files. While these small files may consume minimal actual disk space (bytes or kilobytes each), each one still consumes a full inode. Modern file systems allocate a fixed number of inodes when created, often proportional to the total size of the partition. If this fixed number is exhausted, no new files or directories can be created, even if gigabytes or terabytes of actual disk space remain free.

Common scenarios leading to inode exhaustion in WSL2:

  • Development Caches: Package managers (npm, yarn, pip), build systems, and IDEs can generate millions of temporary or cached files.
  • Session Files: Web applications (PHP, Python, Node.js) often store session data as tiny files in /var/lib/php/sessions or similar directories.
  • Temporary Files: Unmanaged /tmp directories from various processes.
  • Log Files: Especially if logging is verbose and not properly rotated, or if a service is crashing rapidly, creating many small log segments.
  • Docker Build Layers/Intermediate Files: Docker often creates many temporary files during complex build processes or in container volumes if not properly cleaned up.
  • Version Control Systems: Large Git repositories with many small files or rapid changes can sometimes contribute, though less common as a primary cause.
  • Unintentionally copied files: Recursive copies gone wrong, creating deeply nested directories with empty or tiny files.

WSL2 uses a virtual hard disk file (ext4.vhdx) stored on your Windows host. While this file can grow dynamically, the inode allocation within the ext4 filesystem inside WSL2 is determined during its initial creation and resizing. Even if the ext4.vhdx on Windows still has plenty of room to expand, the number of inodes within the ext4 filesystem might be fixed and fully utilized.

Step-by-Step Resolution

The resolution involves identifying directories with a high concentration of small files and safely cleaning them up.

1. Verify Inode Exhaustion

First, confirm that inode exhaustion is indeed the issue.

df -i

If /dev/sdb (or your root filesystem) shows 100% IUse%, proceed.

2. Locate Directories with High Inode Counts

The most effective way to find inode hogs is to count files per directory. Start from the root and drill down.

# Count files (including directories) in each top-level directory
echo "Inodes per top-level directory:"
sudo find / -xdev -printf '%hn' | sort | uniq -c | sort -rh | head -n 20

# Explanation:
# - `find / -xdev`: Search from root, do not cross filesystem boundaries.
# - `-printf '%hn'`: Print the directory name for each found file/directory.
# - `sort`: Sorts the list for `uniq -c`.
# - `uniq -c`: Counts consecutive identical lines (i.e., files per directory).
# - `sort -rh`: Sorts numerically in reverse (highest counts first).
# - `head -n 20`: Shows the top 20 directories.

This command will give you a good starting point, showing which directories contain the largest number of files/inodes. Pay close attention to /var, /usr, /opt, and user home directories (/home/<user>).

3. Clean Up Common Inode Hogs

Once you've identified potential problem areas, target these for cleanup.

a. Clear apt Package Cache

apt caches downloaded .deb files, which can accumulate over time.

sudo apt clean
b. Clear npm/yarn Caches

Node.js development often creates massive caches of tiny files.

# For npm
npm cache clean --force

# For yarn
yarn cache clean --all
c. Prune Docker Resources

Docker can leave behind a significant number of unused images, containers, volumes, and build cache. These often include many layers and temporary files.

# Prune all unused Docker containers, networks, images (dangling and unreferenced), and optionally volumes.
docker system prune -a

# If you also want to remove ALL volumes (use with extreme caution!)
# docker system prune -a --volumes

docker system prune -a removes all stopped containers, all dangling images, all unused images (not just dangling ones), all unused networks, and optionally all build cache. Use with caution, as it will free up significant space and inodes but might remove resources you intended to keep.

d. Manage Log Files

Log files can proliferate, especially during debugging or if a service is misbehaving.

  • Check log sizes:
    sudo du -sh /var/log/* | sort -rh | head -n 10
  • Clear specific log files (use with caution):
    # Truncate a large log file without deleting it (useful for active logs)
sudo truncate -s 0 /var/log/syslog
# Example for other logs:
# sudo truncate -s 0 /var/log/nginx/access.log
# sudo truncate -s 0 /var/log/nginx/error.log
  • Run logrotate manually (forces rotation and compression):
    sudo logrotate -f /etc/logrotate.conf
    This command forces the rotation of all configured logs, potentially creating new, smaller files and compressing old ones, freeing up inodes.
e. Clean Temporary Directories

Check /tmp, /var/tmp, and user-specific temporary directories.

# Remove old files from /tmp (older than 7 days)
sudo find /tmp -type f -atime +7 -delete
sudo find /tmp -type d -empty -delete # Delete empty directories

# Same for /var/tmp
sudo find /var/tmp -type f -atime +7 -delete
sudo find /var/tmp -type d -empty -delete

# Consider specific application temp directories (e.g., PHP sessions)
# For PHP sessions (example path, verify yours):
# sudo find /var/lib/php/sessions -type f -mmin +$(cat /etc/php/*/fpm/php.ini | grep "session.gc_maxlifetime =" | awk '{print $3}') -delete

Be extremely careful when using rm -rf or find ... -delete in system directories. Always double-check your path and command before executing. find ... -delete is generally safer as it only deletes files/directories that match the criteria.

4. Find and Delete Orphaned/Empty Files or Directories

Sometimes processes create many empty files or directories that persist.

# Find and delete empty files older than 30 days
sudo find / -xdev -type f -size 0 -atime +30 -delete

# Find and delete empty directories (use with caution, can remove useful empty scaffolding)
sudo find / -xdev -type d -empty -delete

The find / -xdev command can take a very long time on large filesystems. Start with specific problematic directories identified in step 2.

5. Analyze Specific Application/User Directories

If the general cleanup doesn't resolve the issue, you'll need to dig deeper into the directories identified in step 2.

  • Navigate to the suspicious directory (e.g., /var/www/html/mysite/cache).
  • Run the inode count command again within that directory:
    cd /path/to/suspicious/directory
sudo find . -xdev -printf '%hn' | sort | uniq -c | sort -rh | head -n 20
  • Based on the output, investigate what application or process is creating these files and determine if they can be safely deleted or if the application needs its cache/temporary file management reconfigured.

6. Compact the WSL2 Virtual Disk (Optional, for general disk space cleanup)

While not directly related to inode exhaustion (as inodes are a filesystem metadata issue, not raw disk space), after a large cleanup, your ext4.vhdx file on Windows might still be large. You can reclaim this physical space on your Windows host.

  1. Shut down all WSL2 instances: Open PowerShell or Command Prompt on Windows:

    wsl --shutdown

  2. Compact the ext4.vhdx disk: Navigate to your WSL2 distribution's storage path. The default is usually: %LOCALAPPDATA%Packages<DistroName>LocalState e.g., C:Users<YourUser>AppDataLocalPackagesCanonicalGroupLimited.UbuntuonWindows_...LocalState

    Open Command Prompt or PowerShell as Administrator and run diskpart:

    diskpart

    Within diskpart, execute:

    select vdisk file="C:Users<YourUser>AppDataLocalPackagesCanonicalGroupLimited.UbuntuonWindows_...LocalStateext4.vhdx"
compact vdisk
exit

    Replace the placeholder path with the actual path to your ext4.vhdx file. This process can take some time depending on the disk size and amount of free space.

7. Monitor Inode Usage

After cleanup, regularly monitor your inode usage, especially if you run services that generate many small files.

# Check current inode usage
df -i

# Set up monitoring (e.g., a simple cron job or systemd timer)
# Example cron entry to check inodes and email if over 90% (requires mail utilities)
# 0 * * * * df -i | grep /dev/sdb | awk '{print $5}' | sed 's/%//' | xargs -I {} bash -c 'if [ {} -ge 90 ]; then echo "WARNING: WSL2 Inodes at {}%"; fi' | mail -s "WSL2 Inode Warning" [email protected]

By systematically identifying and clearing inode-heavy directories, you can resolve the "No space left on device" errors in your WSL2 Ubuntu environment and maintain a healthy filesystem.