Debian 12 Disk Full: Troubleshooting & Resolving Inode Exhaustion (`df -i` 100%)

Diagnose and resolve inode exhaustion on Debian 12 Bookworm when `df -h` shows free space but services fail. Prevent critical system outages.

Diagnose and resolve inode exhaustion on Debian 12 Bookworm when `df -h` shows free space but services fail. Prevent critical system outages.

When your Debian 12 server starts behaving as if its disk is completely full – applications failing to write logs, new files failing to create, and web services returning "No space left on device" errors – it's natural to first check disk usage with df -h. However, if df -h reports ample free space, you're likely facing a less common but critical issue: inode exhaustion. This guide will walk you through diagnosing and resolving full inodes on your Debian 12 Bookworm system.

Symptom & Error Signature

The primary symptom is that your system, or specific applications, report "No space left on device" errors, or fail to create new files, even though standard disk space checks indicate free space.

You'll typically observe the following:

  1. df -h output (shows available space):

    df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       100G   20G  80G  20% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           789M  1.1M  788M   1% /run

    Notice here, / has 80G free and only 20% utilization.

  2. df -i output (shows 100% inode usage):

    df -i
Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/sda1      65536000 65536000      0  100% /
tmpfs           1012546    744 1011802    1% /dev/shm
tmpfs           1012546    792 1011754    1% /run

    Crucially, /dev/sda1 (mounted on /) shows 100% IUse% (inode usage). This is the smoking gun.

  3. Application Error Examples:

    • Nginx/PHP-FPM: Error logs may show messages like:
      2026/07/22 14:30:01 [crit] 1234#1234: *5432 open() "/var/cache/nginx/temp/1/00/0000000001" failed (28: No space left on device)
PHP Warning: Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (.../sess_xxxxxxxxxxxxx) in Unknown on line 0
    • System Logs (journalctl -xe or /var/log/syslog):
      systemd[1]: Failed to start Cleanup of Temporary Directories.
kernel: EXT4-fs (sda1): VFS: Can't create inode for new file
    • Package Manager (apt):
      E: mkstemp /tmp/apt.conf.XXXXXXXXXX failed (28: No space left on device)
E: Could not create temporary file for /var/cache/apt/archives/partial/php_8.2-1~deb12u1_all.deb - mkstemp (28: No space left on device)

Root Cause Analysis

An inode (index node) is a fundamental data structure in a Unix-like file system (such as ext4, XFS, etc.) that describes a file system object like a file or a directory. Each file or directory on your system requires exactly one inode. An inode stores metadata about the file, such as:

  • Owner and group IDs
  • Permissions
  • Timestamps (creation, modification, access)
  • File type (regular file, directory, symlink, block device, etc.)
  • The actual disk block addresses where the file's data is stored.

When a file system is created, a fixed number of inodes are allocated. This number determines the maximum number of files and directories that can exist on that file system.

Inode exhaustion occurs when:

  1. Millions of small files are created. This is the most common scenario. Each of these tiny files, regardless of its size (even 0 bytes), consumes one inode. If you have millions of tiny files, you can exhaust all available inodes long before you run out of actual disk space.
  2. Common culprits include:
    • Session files: Web applications (PHP, Node.js) often store session data as individual files.
    • Cache files: Application caches (Nginx, Apache, PHP-FPM, web frameworks like Magento, WordPress) can generate countless temporary or cached files.
    • Mail queues: Mail servers (Postfix, Exim) temporarily store individual emails as files in a queue directory.
    • Temporary files: System-wide or application-specific /tmp directories can accumulate many small, uncleaned files.
    • Docker artifacts: Old Docker layers, images, and container ephemeral data can create numerous small files.
    • Log files: If not properly rotated, applications can create excessively granular or numerous log files.
    • Package manager caches: apt and other package managers can accumulate many small package lists and temporary files.

The system cannot create any new files, directories, or even symbolic links once all inodes are consumed, leading to the "No space left on device" error, even if df -h shows plenty of free blocks.

Step-by-Step Resolution

Follow these steps carefully to identify and clear the inode hogs on your Debian 12 system.

1. Verify Inode Exhaustion

Confirm the issue by checking both disk space and inode usage.

df -h   # Check human-readable disk space
df -i   # Check inode usage

If df -i shows 100% usage for your root filesystem (or a relevant mount point), proceed.

2. Identify Inode Hogs

The goal is to find which directories contain the largest number of files. We'll use find and wc -l (word count – lines) to count files.

The find command can be resource-intensive on a struggling system. Use nice to lower its priority if performance is critical, and screen or tmux in case your SSH session disconnects. Use sudo as needed to ensure you can access all directories.

First, identify the top-level directories that contain the most files:

echo "Counting files in top-level directories..."
for i in /*; do
  if [ -d "$i" ]; then
    printf "%-20s %sn" "$i:" "$(find "$i" -xdev -type f | wc -l)"
  fi
done | sort -rn -k2 | head -10

This command iterates through root directories, counts all regular files (-type f) within them, prints the count, and then sorts to show the top 10. The -xdev flag is crucial to prevent find from crossing into other filesystems (e.g., separate /boot or mounted network shares), ensuring you're only counting files on the affected partition.

Once you've identified a suspicious directory (e.g., /var, /tmp, /home/user/cache), drill down into it. Let's assume /var was identified as having a high count.

echo "Counting files in subdirectories of /var..."
for i in /var/*; do
  if [ -d "$i" ]; then
    printf "%-30s %sn" "$i:" "$(find "$i" -xdev -type f | wc -l)"
  fi
done | sort -rn -k2 | head -10

Repeat this process, drilling down into the directories that show the highest file counts. Common paths to investigate further are /var/lib/docker, /var/cache, /var/tmp, /var/www, /var/log, /var/spool.

Alternatively, you can use find to get the counts directly for subdirectories within a specific path:

# Example: Find top 10 directories with most files under /var/lib/docker
sudo find /var/lib/docker -xdev -type f -print0 | xargs -0 -n 1 dirname | sort | uniq -c | sort -rn | head -10

This command finds all files, extracts their directory names, counts occurrences of each directory, and lists the top 10 directories containing the most files.

3. Safely Clear Inode Hogs

Once you've pinpointed the offending directories, you need to safely remove the files. Be extremely cautious and double-check your commands. Deleting the wrong files can lead to data loss or system instability.

Always verify the contents of directories before mass deletion. Consider backing up critical data if unsure. For production systems, schedule downtime if possible.

a. Log Files (/var/log) Excessive log files can quickly fill inodes.

  • Rotate logs with logrotate: Ensure logrotate is properly configured for all applications.
    sudo logrotate -f /etc/logrotate.conf # Force rotation of all logs
  • Manually delete old log files:
    # Delete log files older than 7 days in /var/log
sudo find /var/log -type f -name "*.log" -mtime +7 -delete
# Delete compressed log archives older than 30 days
sudo find /var/log -type f -name "*.gz" -mtime +30 -delete
# Delete specific application logs (e.g., Nginx access logs)
sudo find /var/log/nginx -type f -name "access.log.*" -mtime +7 -delete

b. Temporary Files (/tmp, /var/tmp) These directories are for temporary storage.

# Delete files in /tmp older than 1 day (use with extreme caution on active systems)
sudo find /tmp -type f -atime +1 -delete
# Delete files in /var/tmp older than 7 days
sudo find /var/tmp -type f -atime +7 -delete

Be very careful with /tmp as active applications might be using temporary files. systemd-tmpfiles-clean.service usually handles this. For immediate relief, clearing older files is generally safe.

c. Web Application Cache/Session Files (/var/www, /var/cache) Many web applications generate vast numbers of cache or session files.

  • Identify specific application directories:

    • /var/www/html/your-app/cache
    • /var/www/html/your-app/sessions
    • /var/cache/nginx (if Nginx caching is enabled)
    • /var/cache/apache2 (if Apache caching is enabled)

  • Example for Nginx cache:

    sudo rm -rf /var/cache/nginx/*
# Or for specific proxy caches
sudo rm -rf /path/to/nginx/proxy_cache_dir/*

  • Example for PHP session files:

    sudo find /var/lib/php/sessions -type f -atime +1 -delete
# Or, if your PHP applications use custom session paths, adjust accordingly.
# Example: sudo find /var/www/html/my-app/sessions -type f -atime +1 -delete

  • Example for a specific application's cache (e.g., WordPress/Magento/Laravel):

    # Example: For a WordPress installation
sudo find /var/www/html/wordpress/wp-content/cache -type f -atime +1 -delete
# Example: For a Laravel installation
sudo rm -rf /var/www/html/laravel-app/bootstrap/cache/*
sudo rm -rf /var/www/html/laravel-app/storage/framework/cache/data/*

    Always consult your application's documentation for safe cache clearing methods.

d. Docker Data (/var/lib/docker) Docker can accumulate many unused images, containers, and volumes, creating numerous small files.

# Clean up unused Docker objects: stopped containers, unused networks, dangling images
sudo docker system prune
# Clean up all unused Docker objects including unused images, volumes and build cache
sudo docker system prune -a
# Clean up unused volumes (use with caution, can delete important data)
sudo docker volume prune

docker system prune -a can delete data that might be needed later (e.g., build cache). Use it judiciously, especially docker volume prune. Ensure you don't delete volumes containing persistent data.

e. APT Cache (/var/cache/apt) The package manager's cache can also accumulate files.

sudo apt clean # Removes downloaded package files (.deb)
sudo rm -rf /var/lib/apt/lists/* # Clears package lists cache, will be rebuilt on next apt update

f. Mail Queues (/var/spool/postfix, /var/spool/exim4) If your server sends a lot of email, especially failed deliveries, mail queues can grow large.

  • Postfix:
    sudo postsuper -d ALL # Deletes all messages from the Postfix queue

    This irrevocably deletes all pending emails in the queue. Only use if you understand the implications.

  • Exim:
    sudo exim -qff # Force delivery attempt for all messages
# Or to delete all messages:
sudo find /var/spool/exim4/input -type f -delete

    This irrevocably deletes all pending emails in the queue.

After performing cleanup, re-check your inode usage:

df -i

You should see a significant drop in IUse%. If not, repeat step 2 and 3 to find more inode hogs.

4. Monitor Inode Usage

Once resolved, set up proactive monitoring to prevent future recurrences.

  • Manual Check: Use watch -n 60 "df -i" to monitor inode usage every 60 seconds.
  • Monitoring Tools: Integrate df -i output into your existing monitoring stack (e.g., Prometheus Node Exporter, Nagios, Zabbix). Look for metrics related to node_filesystem_inodes_free and node_filesystem_inodes_total.

5. Prevent Future Inode Exhaustion

Proactive measures are key to maintaining a healthy system.

  • Configure logrotate properly: Review /etc/logrotate.d/ for all applications, ensuring logs are rotated, compressed, and old ones are removed. Create custom logrotate configurations for applications writing logs outside standard locations.
    # Example /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    daily
    missingok
    rotate 7
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    prerotate
        if [ -d /run/systemd/system ]; then
            systemctl reload nginx.service > /dev/null 2>&1 || true
        else
            /etc/init.d/nginx reload > /dev/null 2>&1 || true
        fi
    endscript
}
  • Application-Specific Cleanup:
    • Web Server Caches: Configure Nginx/Apache cache management settings (e.g., proxy_cache_path ... inactive=1d max_size=10g manager_frequency=1h).
    • Application Caches: Implement application-level cron jobs or built-in mechanisms to clear old session files, temporary files, and cache entries.
  • Regular Docker Pruning: Schedule docker system prune (without -a for caution) as a weekly or monthly cron job.
    # Example cron job to run weekly
0 0 * * 0 /usr/bin/docker system prune -f > /dev/null 2>&1
  • Monitor Mail Queues: Regularly check mail server queues for excessive messages.
  • Filesystem Choice: For new deployments that anticipate storing a huge number of small files (e.g., object storage backends, heavily cached applications), consider using filesystems like XFS, which can dynamically allocate inodes or generally handle larger inode counts more efficiently than ext4 in some configurations.

Changing a filesystem type (e.g., from ext4 to XFS) requires reformatting the partition, which will erase all data. This is a destructive operation and should only be considered for new systems or when you can completely back up, reformat, and restore data.

By understanding the nature of inodes and implementing robust cleanup and monitoring strategies, you can prevent future inode exhaustion issues and maintain the stability and performance of your Debian 12 servers.