Resolving Inode Exhaustion: ‘No Space Left on Device’ on Ubuntu 22.04 LTS

Diagnose and fix critical 'no space left on device' errors due to inode exhaustion on Ubuntu 22.04 LTS, even when df -h shows free space.

A common and often perplexing issue for Linux system administrators is encountering "No space left on device" errors, even when df -h reports ample free disk space. This scenario almost invariably points to inode exhaustion, meaning your filesystem has run out of available inodes rather than actual data blocks. This guide provides a highly technical, step-by-step approach to diagnose, resolve, and prevent inode exhaustion on Ubuntu 22.04 LTS servers, a critical skill for maintaining stable web hosting environments.

Symptom & Error Signature

When your system runs out of inodes, various applications and system processes will fail, often with generic "No space left on device" errors. Users might experience:

• Web server errors (e.g., Nginx failing to write temporary files, PHP session errors).
• Inability to create new files or directories.
• Mail server issues (e.g., inability to receive new mail).
• Application logs failing to write.
• System updates failing.
• SSH login issues due to inability to write user session files.

The key diagnostic output confirming inode exhaustion comes from the df -i command:

df -i

Example Output (Illustrating Inode Exhaustion):

Filesystem     Inodes IUsed IFree IUse% Mounted on
tmpfs           4.0M   875  4.0M    1% /run
/dev/sda1        1.2M  1.2M     0  100% /
tmpfs           4.0M     1  4.0M    1% /dev/shm
tmpfs           4.0M     4  4.0M    1% /run/lock
tmpfs           4.0M    18  4.0M    1% /run/user/1000

Notice /dev/sda1 (your root filesystem or primary partition) showing IUse% at 100% and IFree at 0, despite df -h potentially showing free gigabytes.

Other common error messages observed in application logs or terminal:

touch: cannot touch 'newfile': No space left on device
mkdir: cannot create directory 'newdir': No space left on device
nginx: [emerg] open() "/var/lib/nginx/tmp/client_body/0000000001" failed (28: No space left on device)
PHP Warning: Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php/sessions) in Unknown on line 0

Root Cause Analysis

An inode (index node) is a data structure on a Unix-style filesystem that stores metadata about a file or directory. This includes attributes like file type, permissions, owner, group, size, timestamps, and pointers to the actual data blocks on the disk. Every file and directory on a filesystem consumes one inode.

A fixed number of inodes are created when a filesystem is initially formatted. While modern filesystems generally allocate a generous number of inodes, they are finite. Inode exhaustion occurs when a system creates an extremely large number of very small files, consuming all available inodes before the actual disk space (data blocks) is fully utilized.

Common culprits for inode exhaustion in web hosting and server environments include:

• Session files: PHP or other application session files, especially when not properly garbage collected, can accumulate quickly in directories like /var/lib/php/sessions.
• Cache files: Web server caches (e.g., Nginx fastcgi_cache, Proxy cache), application-specific caches (e.g., Composer, npm, WordPress/Joomla caches), or CDN cache directories can generate millions of tiny files.
• Temporary files: Uncleaned temporary directories (/tmp, /var/tmp), especially those created by long-running processes or failed operations.
• Mail queues: Large numbers of small emails stuck in the mail spool (e.g., /var/spool/postfix/maildrop, /var/mail).
• Log files: While logrotate usually handles this, misconfigurations can lead to an explosion of small log files, especially from frequently executed cron jobs or rogue scripts.
• Docker artifacts: In older Docker versions or specific configurations, Docker build caches or overlay2 layers can sometimes accumulate many small files, although docker system prune usually mitigates this.
• Broken symlinks/empty files: While less common, a script endlessly creating tiny files or broken symlinks can contribute.

Step-by-Step Resolution

> [!WARNING]

Proceed with extreme caution when deleting files, especially as root. Incorrectly deleting system files can render your system unbootable. Always verify paths and commands before execution. Consider creating a snapshot or backup if operating in a virtualized environment.

1. Verify Inode Usage

First, confirm that inode exhaustion is indeed the problem.

df -i

Look for any filesystem reporting IUse% near or at 100%. The affected filesystem is where you need to focus your efforts. Typically, this is the root filesystem /.

2. Identify Top-Level Directories Consuming Most Inodes

To pinpoint the directories containing the most files (and thus consuming the most inodes), you can use a combination of find and wc -l or other tools. This method focuses on counting files per directory.

Method A: Count files in top-level directories

This command iterates through top-level directories on the root filesystem (/) and counts the number of files within each, ignoring other mount points.

echo "Counting files in top-level directories (this may take a while):"
for i in /*; do
    if [ -d "$i" ]; then
        count=$(find "$i" -xdev -type f 2>/dev/null | wc -l)
        echo "$count $i"
    fi
done | sort -rh | head -n 10

Example Output:

Counting files in top-level directories (this may take a while):
1234567 /var
876543 /usr
12345 /opt
...

This will give you the primary target directory (e.g., /var).

Method B: Drill down with find (more precise for specific paths)

Once you have a target (e.g., /var), you can drill down further. This command lists the top 20 subdirectories (two levels deep) within /var that contain the most files.

find /var -xdev -maxdepth 2 -type d -print0 | xargs -0 bash -c 'for d; do count=$(find "$d" -xdev -type f 2>/dev/null | wc -l); echo "$count $d"; done' _ | sort -rh | head -n 20

> [!TIP]

The -xdev option is crucial. It prevents find from traversing into other mounted filesystems (e.g., /boot, /home if they are separate partitions), ensuring you only count inodes on the problematic filesystem.

Common directories to investigate based on the above output:

• /var/lib/php/sessions (PHP session files)
• /var/tmp or /tmp (Temporary files)
• /var/log (Log files, especially if logrotate is misconfigured)
• /var/cache/nginx (Nginx cache files)
• /var/lib/docker/overlay2 or /var/lib/docker/tmp (Docker artifacts)
• /var/spool/postfix/maildrop or similar (Mail queue)
• User home directories, especially if containing large node_modules or build caches for web applications.

3. Safely Clear Inode Hogs

Once you've identified the problematic directory, proceed with caution.

a. Clear PHP Session Files

If /var/lib/php/sessions is identified as a major inode consumer:

# List some of the oldest session files (optional, for inspection)
ls -lt /var/lib/php/sessions | tail -n 20

# Delete session files older than X days (e.g., 24 hours = 1 day)
# This command finds files in /var/lib/php/sessions that are regular files (-type f)
# and have not been accessed in the last 1 day (-atime +1).
# Modify +1 to a higher number if you need to retain sessions longer.
find /var/lib/php/sessions -type f -atime +1 -delete

# Immediately verify inode usage again
df -i

> [!IMPORTANT]

Deleting current PHP session files will log out active users from web applications. Only proceed if you understand the impact or during a maintenance window. PHP's built-in garbage collection usually handles this, but a full disk prevents it from running.

b. Clear Temporary Files

Directories like /tmp and /var/tmp are common culprits.

# Delete files older than 7 days in /tmp (adjust +7 as needed)
find /tmp -type f -atime +7 -delete

# Delete files older than 7 days in /var/tmp (adjust +7 as needed)
find /var/tmp -type f -atime +7 -delete

# Also check for empty directories that might be consuming inodes
find /tmp -type d -empty -delete
find /var/tmp -type d -empty -delete

df -i

c. Clear Nginx Cache

If Nginx cache directories (e.g., /var/cache/nginx) are consuming inodes:

# List cache directories
ls -d /var/cache/nginx/*

# > [!WARNING]
# > Clearing Nginx cache will temporarily increase server load as content is re-cached.
# > Avoid deleting cache files that are actively being written to.
# Stop Nginx to ensure files are not in use (optional, but safest)
# sudo systemctl stop nginx

# Delete all files within the cache directories (adjust path if needed)
# Example: If cache is in /var/cache/nginx/proxy_cache
sudo find /var/cache/nginx/proxy_cache -type f -delete

# If you stopped Nginx, restart it
# sudo systemctl start nginx

df -i

Alternatively, nginx itself can be configured to purge old cache entries. For an immediate fix, direct deletion is faster.

d. Clear Docker Build Cache and System Prune

Docker can generate numerous temporary files and layers.

# Prune all stopped containers, unused networks, dangling images, and build cache
# This is generally safe and recommended for maintenance.
docker system prune -a --volumes

# Prune build cache specifically
docker builder prune

df -i

> [!IMPORTANT]

docker system prune -a --volumes will remove ALL unused Docker data, including volumes not associated with any container. Review the output carefully before confirming.

e. Manage Log Files

While logrotate is designed for this, misconfiguration can lead to problems.

# Check logrotate configuration
cat /etc/logrotate.conf
ls /etc/logrotate.d/

# Manually force logrotate to run (can help if it's stuck)
sudo logrotate -f /etc/logrotate.conf

# If specific directories like /var/log/apache2 or /var/log/nginx
# have millions of small files (e.g., one file per request),
# inspect the configuration that creates them.
# To clean up old log files in a specific directory (e.g., /var/log/myapp/):
# find /var/log/myapp/ -type f -name "*.log" -mtime +30 -delete

df -i

> [!WARNING]

Do not delete active log files that are currently being written to by applications without first stopping those applications or redirecting their logging.

4. Post-Cleanup Verification and Monitoring

After deleting files, immediately check the inode usage again:

df -i

You should see a significant decrease in IUse% for the affected filesystem. Continue to monitor your system's inode usage, especially after applying these fixes.

watch -n 5 df -i

5. Implement Preventative Measures

To avoid future inode exhaustion:

• Configure Logrotate Properly: Ensure all application logs are managed by logrotate. Review /etc/logrotate.d/* configurations.
• Manage PHP Session Garbage Collection: Verify session.gc_probability, session.gc_divisor, and session.gc_maxlifetime in your php.ini (or respective FPM pool configurations). Ensure a cron job or systemd timer is active to run the PHP garbage collector.
• Regular Docker Pruning: Schedule docker system prune -a --volumes or docker builder prune via a cron job or systemd timer to run periodically.
• Clear Temporary Files with Cron: Create a cron job to periodically clean /tmp and /var/tmp for files older than a certain age.

  # Example cron entry (run daily at 2 AM)
  # 0 2 * * * find /tmp /var/tmp -type f -atime +7 -delete > /dev/null 2>&1
  

• Application-Specific Cache Management: Consult documentation for web applications (WordPress, Laravel, etc.) to configure their internal cache cleaning mechanisms.
• Filesystem Design: For systems frequently creating many small files (e.g., image processing, content delivery), consider a separate partition or a dedicated filesystem (e.g., XFS) for those directories. XFS, unlike EXT4, does not have a fixed number of inodes at creation but allocates them dynamically as needed (up to the capacity of the 64-bit inode number field), making it more resilient to inode exhaustion, though it can still occur if improperly configured or pushed to extremes.
• Review Code/Scripts: Identify and fix any custom scripts or applications that might be generating an excessive number of small files without proper cleanup.

By systematically diagnosing and resolving inode exhaustion and implementing preventative measures, you can ensure the long-term stability and performance of your Ubuntu 22.04 LTS servers.