Troubleshooting ‘No Space Left on Device’ due to Inode Exhaustion on Ubuntu 20.04 LTS

Resolve critical 'out of inodes' errors on Ubuntu 20.04 LTS, causing disk space full issues even with free capacity. Learn to diagnose, clean, and prevent inode exhaustion.

A frustrating and often misunderstood issue in Linux system administration is encountering "No space left on device" errors, even when df -h reports ample free disk space. This peculiar situation almost invariably points to an exhaustion of inodes, not physical storage blocks. As an experienced Systems Administrator, understanding and resolving inode-related problems is crucial for maintaining stable web hosting environments and applications. This guide will walk you through diagnosing, resolving, and preventing inode exhaustion on Ubuntu 20.04 LTS.

Symptom & Error Signature

When your system runs out of inodes, services may fail to start, web applications may throw errors, and even basic terminal commands like creating new files or directories will fail. The key symptom is the "No space left on device" message appearing in logs or directly in your terminal, despite df -h showing available capacity.

Here's what you might see:

1. df -h showing available disk space:

   root@webserver:~# df -h
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/sda1       100G   20G   75G  21% /
   udev            3.9G     0  3.9G   0% /dev
   tmpfs           797M  1.4M  796M   1% /run
   /dev/sdb1        50G   10G   38G  21% /var/www
   

   Notice Use% is low, yet problems persist.

2. df -i showing 100% inode usage:

   root@webserver:~# df -i
   Filesystem       Inodes  IUsed    IFree IUse% Mounted on
   /dev/sda1       6553600 6553600        0  100% /
   udev            1012316     423  1011893    1% /dev
   tmpfs            202633     713   201920    1% /run
   /dev/sdb1       3276800  500000  2776800   16% /var/www
   

   This is the definitive indicator of inode exhaustion.

3. Application or System Errors:

   mkdir: cannot create directory 'test': No space left on device
   

   nginx: [emerg] unlink() "/var/cache/nginx/client_temp/0000000001" failed (28: No space left on device)
   

   PHP Warning: Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php/sessions) in Unknown on line 0
   

   systemd[1]: Failed to start Nginx HTTP Server.
   

Root Cause Analysis

Understanding the root cause of inode exhaustion requires a brief dive into how Linux filesystems work.

• What are Inodes? An inode (index node) is a fundamental data structure in a Unix-style filesystem that describes a filesystem object such as a file or a directory. Each inode stores the metadata about a file, including its type, permissions, owner, group, size, creation time, modification time, and disk block addresses where the file's data is actually stored. Every file and directory on your filesystem consumes exactly one inode.

• Inode vs. Disk Space: When a filesystem is created (e.g., using mkfs.ext4), a fixed number of inodes are allocated. This number is typically determined by the filesystem size and a "bytes-per-inode" ratio. If you create many very small files, you can exhaust all available inodes before you run out of actual disk block space. Once all inodes are used, the system cannot create any new files or directories, even if there are gigabytes of free space remaining, leading to the "No space left on device" error.

• Common Culprits for Inode Exhaustion:

  • Web Server Caches: Nginx fastcgi_cache or proxy_cache storing millions of tiny cached files.
  • PHP Session Files: Uncontrolled or improperly garbage-collected PHP session files in /var/lib/php/sessions.
  • Temporary Files: Applications creating excessive temporary files in /tmp or application-specific temporary directories without proper cleanup.
  • Excessive Logging: Applications generating a huge number of tiny log files, or unrotated log files accumulating.
  • Build Artifacts: CI/CD pipelines or development environments leaving many small intermediate files from builds.
  • Email Queues: Mail servers with massive email queues containing many small message files (e.g., Postfix/Exim queues).
  • User Uploads: Applications allowing users to upload many tiny files (e.g., avatars, thumbnails).

Step-by-Step Resolution

Follow these steps to diagnose, clean up, and prevent inode exhaustion on your Ubuntu 20.04 LTS server.

1. Verify Inode Usage

First, confirm that inode exhaustion is indeed the problem.

df -h
df -i

A df -i output showing 100% usage for a critical filesystem (like /) is a clear indication of inode exhaustion.

2. Identify Inode Hogs

The next critical step is to find where these millions of tiny files are located. We'll use find and du.

# Start by identifying top-level directories consuming the most inodes
echo "Inode usage per top-level directory:"
for i in /var /home /opt /usr /tmp; do
    echo "$i: $(sudo find $i -xdev -printf '%hn' 2>/dev/null | wc -l) files"
done

This initial check gives you a hint. Typically, /var or /tmp are the biggest culprits in hosting environments.

Now, dive deeper into the suspicious directories. For example, if /var showed a high count:

echo "Top 20 directories with most files in /var:"
sudo find /var -xdev -printf '%hn' 2>/dev/null | sort | uniq -c | sort -rh | head -n 20

The find /path -xdev option ensures that find does not traverse into other mounted filesystems (e.g., Docker volumes, separate /home partitions), keeping the search within the target filesystem.

Common locations to investigate further:

• /var/lib/php/sessions (PHP session files)
• /var/cache/nginx (Nginx cache files)
• /var/tmp and /tmp (Temporary files)
• /var/log (Log files)
• /var/lib/docker/overlay2 (Docker layers, if container runtime is not cleaning up effectively)
• /var/spool/postfix or /var/spool/exim (Mail queues)

3. Clean Up Inode Hogs

Once you've identified the directory responsible, proceed with caution to clean up. Always double-check your find commands before executing with -delete.

3.1. PHP Session Files

PHP session files are a very common cause of inode exhaustion if not properly garbage collected.

# Check the count of session files
sudo ls -1 /var/lib/php/sessions | wc -l

# View modification times of a few files to gauge age
sudo ls -la /var/lib/php/sessions | head

To clean up old session files (e.g., older than 4 hours):

sudo find /var/lib/php/sessions -type f -mmin +240 -delete

Adjust -mmin carefully. For production systems, ensure session.gc_probability and session.gc_divisor are correctly configured in php.ini to handle garbage collection automatically. Manual deletion should be a temporary fix.

3.2. Nginx Cache Files

Nginx can generate a huge number of tiny files for its proxy_cache or fastcgi_cache.

First, identify your Nginx cache paths from your Nginx configuration files (e.g., proxy_cache_path directive). A common path is /var/cache/nginx.

# Count files in Nginx client temp directory
sudo ls -1 /var/cache/nginx/client_temp | wc -l

# Delete client temporary files older than 1 hour
sudo find /var/cache/nginx/client_temp -type f -mmin +60 -delete

For general Nginx cache cleanup, you can delete files older than a certain duration. Be aware this will impact performance temporarily as Nginx rebuilds the cache.

# Example: Delete Nginx cache files older than 7 days
sudo find /var/cache/nginx -type f -mtime +7 -delete

Drastically deleting all Nginx cache files (sudo find /var/cache/nginx -type f -delete) will clear the entire cache and force Nginx to re-fetch all content, potentially causing a performance hit. Use this only if absolutely necessary and preferably during a maintenance window.

3.3. Temporary Files (/tmp, /var/tmp)

Many applications use /tmp or /var/tmp for temporary data. These directories should be regularly cleaned.

# Delete files in /tmp older than 7 days
sudo find /tmp -type f -atime +7 -delete

# Delete files in /var/tmp older than 30 days
sudo find /var/tmp -type f -atime +30 -delete

Exercise extreme caution when deleting files in /tmp as active processes might be using them. A system reboot often clears /tmp if it's a tmpfs mount (common on systemd systems).

3.4. Log Files (/var/log)

Ensure logrotate is properly configured for all application and system logs. If not, old logs can accumulate.

# Force logrotate to run for all configured logs
sudo logrotate -f /etc/logrotate.conf

# Manually delete old, unrotated logs for a specific application (e.g., myapp)
# This command deletes .log files older than 30 days in /var/log/myapp/
sudo find /var/log/myapp/ -type f -name "*.log" -mtime +30 -delete

3.5. Docker Build Cache/Images/Volumes

If you are using Docker, orphaned volumes, build caches, or unused images can contribute to overall disk and inode usage.

# Clean up unused Docker objects (containers, images, networks, build cache)
# This is a safe first step.
sudo docker system prune

# For a more aggressive cleanup, including all stopped containers and unused images
# WARNING: This is aggressive and will remove ALL unused resources.
# Review carefully before running on production.
sudo docker system prune -a

docker system prune -a can be destructive if not used carefully, as it removes all unused objects. Ensure you understand its implications before executing.

4. Implement Preventative Measures

Once you've cleared the immediate crisis, set up preventative measures to avoid future inode exhaustion.

4.1. Configure Log Rotation for All Applications

Review logrotate configurations in /etc/logrotate.d/ for all applications that generate logs. Ensure they are rotating frequently enough, compressing old logs, and retaining a reasonable number of versions.

Example Nginx logrotate config (/etc/logrotate.d/nginx):

/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    prerotate
        if [ -d /run/systemd/system ]; then
            systemctl reload nginx > /dev/null 2>&1 || true
        else
            /etc/init.d/nginx reload > /dev/null 2>&1 || true
        fi
    endscript
}

4.2. Optimize PHP Session Cleanup

Ensure php.ini is correctly configured for session garbage collection. Look for session.gc_probability, session.gc_divisor, and session.gc_maxlifetime.

Example recommended settings in /etc/php/7.4/fpm/php.ini (or appropriate PHP version):

session.gc_probability = 1
session.gc_divisor = 1000
session.gc_maxlifetime = 1440 # 24 minutes, default for many systems

These settings mean there's a 1/1000 chance that garbage collection will run on any session start, cleaning up sessions older than gc_maxlifetime.

4.3. Nginx Cache Management

Properly configure proxy_cache_path or fastcgi_cache_path directives in your Nginx configuration to manage cache size and expiration.

proxy_cache_path /var/cache/nginx/my_cache levels=1:2 keys_zone=my_cache_zone:10m inactive=7d max_size=10g manager_sleep=0.1s manager_zone_size=200k manager_files=20000;

• inactive=7d: Cache entries not accessed for 7 days will be deleted by the cache manager process.
• max_size=10g: Maximum size of the cache. When exceeded, the cache manager removes the least recently used data.

4.4. Use tmpfs for Temporary Files

Consider mounting /tmp as a tmpfs filesystem. tmpfs filesystems reside entirely in RAM (and swap space if needed), and their contents are automatically cleared on reboot. This effectively manages /tmp and prevents inode accumulation there.

Add the following line to /etc/fstab:

tmpfs /tmp tmpfs defaults,noatime,nosuid,nodev,size=2G 0 0

Adjust size=2G to your system's needs. Then, mount it:

sudo mount -a

Files stored in tmpfs are volatile and will be lost on system reboot. Ensure no critical or persistent data is being stored in /tmp by applications if you implement this.

4.5. Monitor Inode Usage

Implement proactive monitoring for inode usage on critical filesystems. Use tools like Prometheus, Grafana, Zabbix, or Nagios to alert you when inode usage exceeds a predefined threshold (e.g., 80% or 90%).

A simple script for cron-based monitoring:

#!/bin/bash
THRESHOLD=90
FILESYSTEM="/" # Monitor root filesystem
CURRENT_INODES=$(df -i "${FILESYSTEM}" | tail -1 | awk '{print $5}' | sed 's/%//')

if [ "${CURRENT_INODES}" -gt "${THRESHOLD}" ]; then
    echo "CRITICAL: Inode usage on ${FILESYSTEM} is ${CURRENT_INODES}%"
    # Add your notification logic here (e.g., email, Slack webhook)
    # mail -s "Inode Alert on $(hostname)" [email protected] <<< "Inode usage on ${FILESYSTEM} is ${CURRENT_INODES}%"
fi

Save this as /usr/local/bin/check_inodes.sh, make it executable (chmod +x), and add it to your root crontab (sudo crontab -e) to run periodically (e.g., every 5 minutes):

*/5 * * * * /usr/local/bin/check_inodes.sh

4.6. Consider Filesystem Reformat (Last Resort)

If inode exhaustion is a chronic problem due to the initial filesystem creation parameters (e.g., a very large disk with a default high bytes-per-inode ratio), and none of the above measures suffice, a full filesystem reformat might be necessary. This is a destructive operation that requires taking the server offline, backing up all data, recreating the filesystem with a different inode density, and restoring the data.

When creating an ext4 filesystem, the -i or --bytes-per-inode option determines the inode density. A lower number means more inodes will be created. The default is typically 1 inode per 16384 bytes.

Example for a new filesystem on /dev/sdb1:

# This command will create one inode for every 8192 bytes, effectively doubling the default inode count.
sudo mkfs.ext4 -i 8192 /dev/sdb1

Reformatting a filesystem is a destructive operation that will erase all data on the partition. It requires extensive planning, full data backups, and significant downtime. This should only be considered as a last resort after exhausting all other options.