Troubleshooting 'No Space Left on Device: Out of Inodes' Error on Linux

When your Linux system or an application starts reporting “No space left on device,” it’s often an immediate thought that your disk is full. However, a less common but equally critical variant of this error is “No space left on device: out of inodes.” This specific error indicates that while you may have gigabytes of free disk space, the filesystem has run out of inodes – the fundamental data structures required to create new files or directories. This guide will walk you through understanding, diagnosing, and resolving this advanced filesystem issue, ensuring your services return to normal operation.

Symptom & Error Signature

Applications and the operating system will fail to create new files, directories, or even log entries, leading to various operational failures. You might see errors in application logs, web server logs, or directly in your terminal when attempting file operations.

Typical Error Output Examples:

1. General System/Bash Error:

$ touch new_file.txt
touch: cannot touch 'new_file.txt': No space left on device

$ mkdir my_new_directory
mkdir: cannot create directory 'my_new_directory': No space left on device

2. Nginx Error Log (/var/log/nginx/error.log):

2023/10/27 10:30:05 [crit] 12345#12345: *1 open() "/var/cache/nginx/proxy_temp/1/00/0000000001" failed (28: No space left on device) while reading upstream, client: 192.168.1.1, server: example.com, request: "GET /large-file HTTP/1.1", upstream: "http://backend:8080/large-file"

3. Systemd Journal Log (journalctl -xe):

Oct 27 10:31:15 myserver systemd-journald[456]: Failed to write entry (errno 28): No space left on device

4. Application-Specific Logs (e.g., PHP, Python, Java):

PHP Warning: session_start(): Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php/sessions) in /var/www/html/app/index.php on line 123

Root Cause Analysis

To understand why you’re encountering an inode exhaustion error, it’s crucial to grasp what an inode is and how filesystems manage them.

What are Inodes? An inode (index node) is a data structure on a Unix-style filesystem that stores metadata about a file or directory. This metadata includes:

• The file’s type (regular file, directory, symbolic link, etc.)
• Permissions (read, write, execute)
• Owner and group IDs
• Timestamps (creation, modification, last access)
• The size of the file
• Pointers to the disk blocks where the file’s actual data is stored.

Crucially, every single file and directory on a filesystem consumes exactly one inode. When a filesystem is created (e.g., with mkfs.ext4), a fixed number of inodes are allocated. This number is determined based on factors like the filesystem size and the bytes-per-inode ratio specified during creation.

Why Inodes Run Out Before Disk Space: The “out of inodes” error occurs when all allocated inode structures are in use, even if there’s still free space in terms of data blocks. This typically happens in environments that generate a vast number of very small files.

Common Scenarios Leading to Inode Exhaustion:

• Excessive Small Files: This is the most common cause. Examples include:
  • Web Server Caches: Nginx FastCGI, proxy, or micro-cache directories (/var/cache/nginx) can accumulate millions of tiny temporary files.
  • Application Caches: Caches for frameworks (e.g., Symfony, Laravel), Composer, npm, or custom application-generated temporary files that aren’t properly cleaned up.
  • Session Files: PHP session files (/var/lib/php/sessions) if not garbage collected efficiently.
  • Email Spools: Mail servers storing individual emails as small files in user mailboxes (e.g., /var/mail or /home/*/Maildir).
  • Build Artifacts/Temporary Files: CI/CD pipelines, Docker builds, or development environments that create and leave behind numerous temporary files.
  • Version Control Systems: Git repositories with a massive number of small objects, though less common as a primary cause.
• Runaway Processes: Misconfigured applications or scripts that continuously generate temporary files without cleaning them.
• Misconfigured Backups/Snapshots: Some backup solutions might create many hard links or temporary copies that consume inodes.
• Denial of Service (DoS) Attacks: Malicious actors could intentionally flood a system by uploading many tiny files or triggering processes that create excessive temporary data.

Step-by-Step Resolution

Follow these steps carefully to diagnose and resolve inode exhaustion.

1. Verify Inode Usage

First, confirm that the problem is indeed inode exhaustion and not simply a full disk. The df command with the -i option shows inode usage.

df -i

Expected Output Indicating Inode Exhaustion:

Filesystem     Inodes IUsed IFree IUse% Mounted on
udev           497127   478 496649    1% /dev
tmpfs          500000   760 499240    1% /run
/dev/vda1     1048576 1048576      0  100% /   <-- This is the culprit!
tmpfs          500000     1 499999    1% /dev/shm
tmpfs          500000    14 499986    1% /run/lock
tmpfs          500000    16 499984    1% /sys/fs/cgroup
/dev/loop0      65536   65536      0  100% /snap/core/11316
/dev/loop1      65536   65536      0  100% /snap/docker/874

In the example above, /dev/vda1 (mounted at /) shows IUse% at 100%, indicating that all inodes are consumed, even if df -h might show free space.

2. Locate the Directory with Excessive Files

Once you’ve identified the filesystem, you need to find which directory or directories are hoarding the most inodes. We’ll use the du command with the --inodes option for this.

# Start by checking the top-level directories of the problematic filesystem (e.g., /)
sudo du --inodes -ahx / | sort -rh | head -20

• --inodes: Reports directory inode usage rather than disk space usage.
• -a: Displays inodes for all files and directories.
• -h: Human-readable format.
• -x: Stays on the same filesystem (prevents crossing mount points).
• sort -rh: Sorts results in reverse human-readable order, showing the largest inode users first.

This command will show you the inode usage of files and directories starting from the root of the specified filesystem. Look for unusually high numbers in specific directories. Common culprits are /var, /tmp, and /home.

Once you identify a large directory (e.g., /var), drill down further:

sudo du --inodes -ahx /var | sort -rh | head -20

Repeat this process until you pinpoint the exact sub-directory causing the issue (e.g., /var/cache/nginx, /var/lib/php/sessions).

[!NOTE] The du --inodes command can be slow on very large filesystems. If you have a suspicion based on the services running on your server, you can start directly by inspecting those paths.

3. Identify and Remove Unnecessary Small Files

Once you’ve located the problematic directory, the next step is to remove the excessive small files. Be extremely cautious and always understand what you’re deleting.

a. Clear Web Server Caches (Nginx Example)

If /var/cache/nginx is identified as the culprit:

# Check size first
sudo du -sh /var/cache/nginx

# Option 1: Delete all files (can be disruptive)
> [!WARNING]
> Deleting Nginx cache files while Nginx is actively running can cause temporary issues or errors for users. It's often safer to stop Nginx first, clear the cache, then restart.
sudo systemctl stop nginx
sudo rm -rf /var/cache/nginx/*
sudo systemctl start nginx

# Option 2: Delete files older than N days (safer, less disruptive)
sudo find /var/cache/nginx -type f -mtime +7 -delete
# This command deletes files older than 7 days in the Nginx cache directory.

b. Clear PHP Session Files

If /var/lib/php/sessions is the issue:

# Check size first
sudo du -sh /var/lib/php/sessions

# Delete session files older than a certain time (e.g., 24 minutes, which is a common garbage collection interval)
sudo find /var/lib/php/sessions -type f -mmin +24 -delete

[!IMPORTANT] Always check your php.ini for session.save_path to ensure you’re cleaning the correct directory. Also, ensure your PHP garbage collection (GC) settings are correctly configured to prevent this issue from recurring.

c. Prune Docker System Data

In a Docker environment, uncleaned images, containers, volumes, and build caches can consume inodes.

# Display disk usage for Docker objects
sudo docker system df

# Prune all unused Docker data (containers, images, volumes, networks, build cache)
> [!WARNING]
> `docker system prune` is a powerful command that can delete a significant amount of data, including stopped containers and dangling images/volumes. Use with caution, especially `--volumes` which removes anonymous volumes.
sudo docker system prune -a --volumes

d. Manage Systemd Journal Logs

Systemd’s journal can consume many inodes if it stores logs in files and grows unchecked.

# Check current journal disk usage
sudo journalctl --disk-usage

# Limit journal size (e.g., to 500MB)
sudo journalctl --vacuum-size=500M

# Keep logs only for a certain period (e.g., 1 week)
sudo journalctl --vacuum-time=1w

e. Clean Up General Temporary Files

The /tmp directory (if not a tmpfs or auto-cleaned) and application-specific temporary directories can also be culprits.

# Delete files in /tmp older than 1 day
sudo find /tmp -type f -mtime +1 -delete

f. Manage Application Logs

Large numbers of application log files, especially if not handled by logrotate, can consume inodes.

# Example: Delete Nginx access/error logs older than 30 days
sudo find /var/log/nginx -type f -name "*.log" -mtime +30 -delete

# Example: Delete gzipped old logs older than 90 days
sudo find /var/log/nginx -type f -name "*.log.gz" -mtime +90 -delete

[!IMPORTANT] Back up critical logs before deletion if compliance or debugging requires it. Services might need to be restarted (sudo systemctl reload nginx) after manual log deletion if they hold open file handles to the deleted files.

4. Configure Proactive Monitoring & Cleanup

To prevent future inode exhaustion, implement proactive measures.

a. Monitor Inodes

Integrate inode usage into your monitoring system (Prometheus, Grafana, Nagios, Zabbix). Set alerts for when inode usage exceeds a threshold (e.g., 80-90%).

b. Implement Logrotate

Ensure logrotate is correctly configured for all system and application logs that might grow large or numerous. Check existing configurations in /etc/logrotate.d/.

Example /etc/logrotate.d/my_app:

/var/log/my_app/*.log {
    daily
    missingok
    rotate 7
    compress
    delaycompress
    notifempty
    create 0640 www-data www-data
    sharedscripts
    postrotate
        # Adjust service name as needed
        systemctl reload my_app.service > /dev/null 2>&1 || true
    endscript
}

• daily: Rotate logs daily.
• rotate 7: Keep 7 rotated log files.
• compress: Compress rotated logs.
• postrotate: Commands to execute after rotation (e.g., restart/reload application).

c. Automate Cache Cleanup with Cron

Create cron jobs to regularly clean up temporary files and caches that logrotate or application-specific mechanisms don’t handle.

Example: /etc/cron.daily/clean-nginx-cache

#!/bin/bash
# Clean Nginx cache files older than 7 days
find /var/cache/nginx -type f -mtime +7 -delete

Example: /etc/cron.daily/clean-php-sessions

#!/bin/bash
# Clean PHP session files older than 24 hours (1440 minutes)
find /var/lib/php/sessions -type f -mmin +1440 -delete

[!IMPORTANT] Ensure your cron scripts have executable permissions (sudo chmod +x /etc/cron.daily/your-script) and are owned by root (sudo chown root:root /etc/cron.daily/your-script). Test them manually before relying on automation.

5. Consider Filesystem Re-creation (Last Resort)

If inode exhaustion is a chronic problem on a specific filesystem (e.g., a partition designed for large files that now hosts many small ones), and regular cleanup isn’t sufficient, you might need to re-create the filesystem with a higher inode density.

[!WARNING] Re-creating a filesystem will result in complete data loss on that partition. This is a highly destructive operation that requires a full backup of all data on the affected partition and a detailed recovery plan. It should only be performed during a scheduled maintenance window.

During filesystem creation using mkfs, you can specify the bytes-per-inode ratio or the total number of inodes. A smaller bytes-per-inode value (e.g., 4096 bytes/inode, which is 4KB) or a larger explicit number of inodes (-N) will allocate more inodes.

# Example for ext4 (DANGER: DO NOT RUN WITHOUT FULL BACKUP AND UNMOUNTED PARTITION)
# Assuming /dev/sdb1 is the target partition
# 1. Backup all data from /dev/sdb1
# 2. Unmount the partition
# sudo umount /dev/sdb1

# 3. Recreate filesystem with higher inode count (e.g., 1 inode per 4KB data)
# sudo mkfs.ext4 -i 4096 /dev/sdb1

# OR explicitly set the number of inodes (e.g., 10 million inodes)
# sudo mkfs.ext4 -N 10000000 /dev/sdb1

# 4. Mount the partition and restore data

The default for ext4 is typically one inode for every 16KB of data. Reducing this to 4KB or 8KB will quadruple or double the available inodes, respectively.

6. Optimize Application Behavior

Review your application’s code and configuration:

• Minimize Temporary Files: Can temporary files be avoided? Can they be written to a tmpfs (RAM filesystem) if their content is not critical for persistence?
• Streaming: For large files, use streaming where possible instead of writing temporary files to disk.
• Cache Management: Ensure application caches have proper eviction policies based on age or size.
• Docker Optimization: For Docker users, properly configure .dockerignore files and optimize multi-stage builds to prevent excessive build cache layers and temporary files.

By systematically diagnosing the problem and implementing both immediate fixes and long-term preventative measures, you can effectively resolve and avoid “No space left on device: out of inodes” errors.