Reclaiming Disk Space: Troubleshooting and Cleaning Massive Systemd Journald Logs on Ubuntu 20.04 LTS

Resolve critical disk space issues caused by oversized systemd journald logs on Ubuntu 20.04 LTS. Learn to clean logs and configure persistent retention.

Introduction

As a seasoned Systems Administrator or DevOps engineer, you've likely encountered the critical scenario where a server's disk space dwindles rapidly, leading to service degradation or even outages. A common culprit on modern Linux distributions like Ubuntu 20.04 LTS is systemd-journald, the system's logging service. By default, journald can be configured to store logs persistently with minimal size limits, allowing them to consume massive amounts of disk space over time, particularly on busy servers or those experiencing recurring errors.

This guide provides a highly technical, accurate, and step-by-step approach to diagnosing and resolving excessive journald log file growth, reclaiming valuable disk space, and implementing robust retention policies to prevent future occurrences on your Ubuntu 20.04 LTS systems.

Symptom & Error Signature

The primary symptom is a server running low on disk space, often manifesting as applications failing to write data, inability to install new packages, or a slow, unresponsive system. You'll typically observe high disk utilization in your monitoring systems or when executing standard disk usage commands.

# Check overall disk usage
df -h

Expected Output (Illustrative of the problem):

Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           795M  1.1M  794M   1% /run
/dev/sda1        97G   92G  1.5G  98% /   <-- High usage on root partition
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/loop0       68M   68M     0 100% /snap/lxd/27885
/dev/loop1       41M   41M     0 100% /snap/snapd/20290
tmpfs           795M     0  795M   0% /run/user/1000

Investigating further will likely reveal that the /var/log/journal directory is consuming the majority of the used space.

# Check disk usage of the journal directory
sudo du -sh /var/log/journal/

Expected Output (Illustrative):

75G     /var/log/journal/  <-- Indicating a massive journal size

Root Cause Analysis

The systemd-journald service, a core component of the systemd init system, collects and stores logging data from the kernel, initrd, services, and applications. On Ubuntu 20.04 LTS, journald is configured by default for persistent storage, meaning logs are written to /var/log/journal/ and survive reboots.

The primary reasons for massive journald log file growth include:

1. Default Unlimited Retention: Without specific configuration, journald may attempt to keep all persistent logs indefinitely, constrained only by available disk space.
2. Verbose Services/Applications: Applications configured with high logging verbosity (e.g., debug mode), or those experiencing frequent errors and warnings, can rapidly generate large volumes of log data.
3. High Traffic/Activity: Servers handling significant traffic or processing intensive tasks naturally produce more log entries.
4. Misconfigured Log Forwarding: If logs are being generated at a high rate but not effectively forwarded to an external logging system (like ELK stack, Splunk, Graylog), they will accumulate locally.

Understanding these underlying factors is crucial for not just cleaning up, but also for preventing recurrence.

Step-by-Step Resolution

Follow these steps to safely clean up existing journald logs and configure persistent retention policies.

1. Assess Current Journald Disk Usage

First, use journalctl itself to report its current disk usage. This command provides a quick and accurate overview.

sudo journalctl --disk-usage

Example Output:

Archived and active journals take up 75.2G in /var/log/journal.

2. Analyze Log Content (Optional but Recommended)

Before cleaning, it's wise to quickly inspect what is filling the logs. This can help identify problematic applications or recurring errors.

# View the most recent log entries (live tail)
sudo journalctl -f

# View recent errors and warnings
sudo journalctl -p err -p warning -b -n 100

# View logs from a specific service (e.g., nginx)
sudo journalctl -u nginx.service -n 50 --since "2 hours ago"

This analysis might reveal services spamming logs or a specific issue leading to excessive logging. Addressing the root cause of high log generation is as important as cleaning up.

3. Clean Up Existing Journal Files

You can clean up old journal entries using journalctl's vacuuming capabilities. This process is safe and will not interrupt active logging. You can specify a maximum size or an age for retained logs.

Be cautious not to set extremely aggressive vacuum parameters without understanding your auditing and compliance requirements. Deleting logs might remove critical forensic data.

a. Vacuum by Size: Reduce the total size of journal files to a specified limit.

# Reduce journal files to a maximum of 500MB (adjust as needed)
sudo journalctl --vacuum-size=500M

b. Vacuum by Time: Remove all journal entries older than a specified time duration.

# Remove journal entries older than 7 days (adjust as needed)
sudo journalctl --vacuum-time=7d

You can combine these for more precise control. For a quick resolution to reclaim space, start with --vacuum-size. After running the command, verify the disk usage again:

sudo journalctl --disk-usage
sudo du -sh /var/log/journal/
sudo df -h

4. Configure Persistent Journald Log Retention Policies

To prevent future uncontrolled growth, you must configure journald's retention policies in its main configuration file: /etc/systemd/journald.conf.

Always back up configuration files before modifying them: sudo cp /etc/systemd/journald.conf /etc/systemd/journald.conf.bak

Open the configuration file using your preferred text editor (e.g., nano or vim):

sudo nano /etc/systemd/journald.conf

Uncomment and modify the following parameters to set your desired limits. Here's a common and recommended configuration for a web server:

[Journal]
# Uncomment and set these values
Storage=persistent
# Sets the maximum disk space that journal files stored in /var/log/journal/ may use.
# Recommended to set a reasonable limit, e.g., 1G or 2G for web servers.
SystemMaxUse=1G

# Sets the minimum disk space to leave free on the file system where /var/log/journal/ resides.
# Ensures that other system processes are not starved for disk space.
SystemKeepFree=200M

# Sets the maximum size of individual journal files in /var/log/journal/.
# This helps in managing individual file sizes and potentially faster log rotation/reading.
SystemMaxFileSize=100M

# Sets the maximum disk space that journal files stored in /run/log/journal/ may use.
# These are volatile logs that do not survive a reboot. Usually smaller than persistent.
RuntimeMaxUse=100M

# Sets the maximum age of stored journal entries. Entries older than this will be vacuumed.
# Can be combined with SystemMaxUse for double control.
# MaxRetentionSec=30 days

Explanation of Key Parameters:

• Storage=persistent: Ensures logs are stored on disk and survive reboots (this is usually the default and desired setting). Set to auto or volatile if you want logs to be stored only in RAM and cleared on reboot (not recommended for production).
• SystemMaxUse=: The most important parameter. This limits the total size of all persistent journal files in /var/log/journal/. Once this limit is reached, journald will automatically delete the oldest entries. Start with 1G or 2G and adjust based on your server's activity and monitoring.
• SystemKeepFree=: Ensures that journald leaves a specified amount of free disk space on the filesystem containing /var/log/journal/. This acts as a safeguard.
• SystemMaxFileSize=: Limits the size of individual journal files. When a file reaches this size, journald will rotate to a new one. This can help with managing large individual log files.
• RuntimeMaxUse=: Similar to SystemMaxUse, but applies to volatile logs stored in /run/log/journal/ (RAM-based). Keep this much smaller.
• MaxRetentionSec=: An alternative or complementary setting to SystemMaxUse. This specifies the maximum time duration for which entries are kept. For example, 30 days.

Recommended Configuration Example:

[Journal]
Storage=persistent
SystemMaxUse=1G
SystemKeepFree=200M
SystemMaxFileSize=100M
RuntimeMaxUse=100M
# MaxRetentionSec=30 days

After saving your changes, you must restart the systemd-journald service for the new configuration to take effect:

sudo systemctl restart systemd-journald

Restarting systemd-journald does not delete existing logs immediately based on the new SystemMaxUse setting. It only applies the new policy going forward. If you just set SystemMaxUse, you might want to run a journalctl --vacuum-size command again to immediately conform to the new limit, if your current usage is still above it.

5. Verify Configuration and Disk Space

After restarting the service and potentially vacuuming again, re-verify your disk usage.

sudo journalctl --disk-usage
sudo du -sh /var/log/journal/
df -h

You should now see the journald usage significantly reduced and df -h reporting a healthier disk utilization.

6. Monitor Disk Usage (Ongoing)

Even after implementing these changes, it's crucial to continue monitoring your server's disk space, especially the /var/log/journal directory. Tools like Prometheus with node_exporter, Zabbix, or simple cron jobs executing df -h and sending alerts can help you stay on top of disk usage trends.

Consider also implementing centralized logging for critical production environments. Forwarding logs to a dedicated logging solution (e.g., Elastic Stack, Graylog) allows for longer retention, advanced analysis, and reduces the disk burden on individual servers.