Debian 12 Bookworm: Resolving Massive systemd journald Log File Disk Usage

Experiencing low disk space on Debian 12 Bookworm due to systemd journald logs? This guide details how to clean up and configure log retention.

When managing web servers or other critical services, unmanaged log file growth is a common culprit for unexpected disk space exhaustion. On modern Debian systems, systemd-journald is responsible for collecting and storing system and application logs. While incredibly powerful for centralized logging, its default configurations can sometimes lead to log files consuming vast amounts of storage, especially on busy systems or those experiencing frequent errors. This guide will walk you through diagnosing, cleaning up, and permanently configuring journald's behavior on Debian 12 Bookworm to prevent future disk space issues.

Symptom & Error Signature

The primary symptom is a system that gradually, or sometimes rapidly, runs out of disk space on the root filesystem or a dedicated /var partition. This can lead to various service failures, applications crashing, and a generally unstable system state.

You'll typically observe this using standard disk usage tools:

# Check overall disk usage
df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   48G  2.0G  96% /

# Identify the largest directories under /var/log
sudo du -sh /var/log/* | sort -rh | head -n 5
40G     /var/log/journal
2.1G    /var/log/nginx
500M    /var/log/mysql
100M    /var/log/apt
80M     /var/log/syslog

If /var/log/journal is one of the top consumers, you can confirm journald's current disk usage with its built-in utility:

# Check journald's reported disk usage
journalctl --disk-usage
Journals take up 40.0G on disk.

When services start failing due to lack of space, you might see errors like:

No space left on device
Failed to write log message: No space left on device

These errors can appear in application logs, service startup failures, or even when trying to execute simple commands.

Root Cause Analysis

The systemd-journald service, by default, stores logs persistently in /var/log/journal. Unlike traditional syslog setups that relied on logrotate for purging old logs, journald manages its own log retention based on size and time policies.

The core reasons for massive journald log file sizes are:

1. Default Retention Policies: While journald does have default size limits (often 10% of the /var/log filesystem or a hard cap like 4GB/8GB, depending on the systemd version and compile-time options), these limits might be too generous for your specific disk size or logging volume. On Debian 12 with systemd 252, the default SystemMaxUse is typically set to 10% of the partition size (up to a certain max) or 4GB if /var/log/journal is empty and persistent storage is enabled. If you have a very large /var partition, 10% could still be tens or hundreds of gigabytes.
2. High Log Volume:
   • Misconfigured Services: Services that continuously restart due to errors (e.g., nginx, php-fpm, mysql, custom applications) will generate a flood of "failed to start", "exited", or error messages.
   • Verbose Debugging: Debugging levels left enabled on critical services can drastically increase log output.
   • Kernel Messages: Hardware issues, network problems, or certain kernel modules can spam logs with repetitive warnings or errors.
   • System Abuse/Attacks: Failed login attempts (SSH, web apps) can fill logs with authentication failures.
3. Lack of Explicit Configuration: Without explicit SystemMaxUse or MaxRetentionSec settings in /etc/systemd/journald.conf, journald will adhere to its default behaviors, which may not align with your system's resource constraints or compliance requirements.

Step-by-Step Resolution

Follow these steps to clean up existing journald logs and implement a permanent retention policy.

1. Assess Current Journald Disk Usage

Before making any changes, confirm the current state:

# Check total disk usage of journald logs
journalctl --disk-usage

# Verify the actual directory size on disk
sudo du -sh /var/log/journal

This will give you a clear picture of how much space is currently occupied.

2. Clean Up Old Journal Entries (Immediate Relief)

You can immediately reclaim disk space by telling journald to vacuum (delete) old log entries. This provides immediate relief but is not a permanent solution, as logs will continue to accumulate.

These commands will delete old log entries. Ensure you do not need historical logs for auditing or debugging before proceeding.

You can vacuum logs based on size or age:

Clean by Size: Delete entries until the total size is below a specified limit.

# Example: Reduce journal size to 1GB
sudo journalctl --vacuum-size=1G

Clean by Time: Delete entries older than a specified time period.

# Example: Keep only logs from the last 7 days
sudo journalctl --vacuum-time=7d

You can combine both for a more aggressive cleanup. For instance, to keep logs for 7 days, but only up to 500MB, whichever comes first:

sudo journalctl --vacuum-time=7d --vacuum-size=500M

After running the vacuum command, verify the disk usage again:

journalctl --disk-usage
sudo du -sh /var/log/journal

3. Configure Persistent Journald Retention Policies (Long-Term Solution)

For a permanent solution, you need to configure journald's behavior in its configuration file.

1. Edit the journald Configuration File: Open /etc/systemd/journald.conf with your preferred text editor (e.g., nano or vim).

   sudo nano /etc/systemd/journald.conf
   

2. Uncomment and Set Retention Parameters: Locate the relevant lines (they might be commented out with #) and set your desired values. Here are the most important parameters:

   • SystemMaxUse=: The maximum amount of disk space the persistent journal files may consume.
   • SystemKeepFree=: The amount of disk space that journald shall leave free for other uses. If SystemMaxUse is set, this value is primarily used to determine when journald should start cleaning up if the total free space on the filesystem falls below this threshold.
   • SystemMaxFileSize=: The maximum size of individual journal files.
   • MaxRetentionSec=: The maximum time to store journal entries. Entries older than this will be deleted.

   Here's an example configuration that's suitable for many web servers with moderate logging:

   # /etc/systemd/journald.conf
   
   [Journal]
   # Set the maximum total size for all persistent journal files to 2GB
   SystemMaxUse=2G
   
   # Ensure at least 500MB of free space is maintained on the filesystem
   SystemKeepFree=500M
   
   # Set the maximum size for a single journal file to 200MB
   SystemMaxFileSize=200M
   
   # Keep journal entries for a maximum of 30 days
   MaxRetentionSec=30day
   
   # Set storage to persistent (default, but good to be explicit)
   Storage=persistent
   

   SystemMaxUse and SystemKeepFree work together. journald will typically aim to keep the total usage below SystemMaxUse, but will also trigger cleanup if SystemKeepFree threshold is breached. SystemMaxUse often takes precedence. Use units like K, M, G, T for size, and s, min, h, day, week, month, year for time.

3. Save Changes and Restart systemd-journald: After saving the journald.conf file, you must restart the systemd-journald service for the changes to take effect.

   sudo systemctl restart systemd-journald
   

4. Trigger Immediate Cleanup (Optional but Recommended): The new configuration will apply going forward. To immediately apply the new retention policies to existing logs, run journalctl --vacuum without arguments, which will respect the MaxRetentionSec and SystemMaxUse settings defined in the config.

   sudo journalctl --vacuum
   

5. Verify Configuration: Check the current disk usage again to confirm cleanup and that the new settings are being respected.

   journalctl --disk-usage
   

   You can also see the active configuration with systemd-analyze:

   systemd-analyze cat-config systemd/journald.conf
   

4. Identify & Address Log-Spewing Services (Preventative)

While configuring journald limits its footprint, it's crucial to identify what is generating excessive logs. Reducing log volume at the source is the most efficient long-term strategy.

1. View Recent Logs:

   journalctl -r -n 50
   

   This shows the 50 most recent log entries (in reverse chronological order) and can quickly highlight services or issues causing frequent logging.

2. Filter by Priority (Errors/Warnings):

   # Show only error messages since last boot
   journalctl -p err -b
   
   # Show errors and critical messages from the last 24 hours
   journalctl -p warning --since "1 day ago"
   

3. Filter by Systemd Unit: If you suspect a specific service (e.g., nginx, php-fpm, mysql), filter its logs:

   journalctl _SYSTEMD_UNIT=nginx.service --since "1 hour ago" -n 100
   journalctl _SYSTEMD_UNIT=php8.2-fpm.service -p err
   

4. Inspect Kernel Logs: Sometimes, kernel messages due to hardware issues or driver problems can fill the logs.

   journalctl -k -p err
   

   Once you identify a log-spewing service, investigate its configuration (/etc/nginx/nginx.conf, /etc/php/8.2/fpm/pool.d/www.conf, /etc/mysql/my.cnf, etc.). Look for recurring errors, debugging levels, or misconfigurations that cause service restarts. Rectify the underlying problem to prevent future excessive logging.

5. Consider Volatile-Only Storage (Advanced/Specific Use Cases)

In some environments (e.g., ephemeral containers, stateless servers where logs are shipped to a central aggregator), you might not need persistent journald logs at all. If /var/log/journal is empty or non-existent, journald will store logs in /run/log/journal by default, which is a tmpfs filesystem and cleared on every reboot.

To achieve this:

1. Remove Persistent Logs:

   sudo rm -rf /var/log/journal
   

2. Configure journald.conf: Edit /etc/systemd/journald.conf and set Storage=volatile.

   # /etc/systemd/journald.conf
   
   [Journal]
   Storage=volatile
   

3. Restart systemd-journald:

   sudo systemctl restart systemd-journald
   

   Setting Storage=volatile means all logs will be lost on system reboot. This is generally not recommended for critical production servers unless you have a robust external logging solution (e.g., Loki, ELK stack, Splunk) that ships logs off-server in real-time.

By following these steps, you can effectively manage systemd-journald log file sizes on your Debian 12 Bookworm system, ensuring stable operation and preventing disk space exhaustion. Regular monitoring of disk usage and log output remains a best practice for any system administrator.