Resolve critical disk space issues on CentOS Stream and Rocky Linux caused by overgrown systemd journald logs. Learn to reclaim storage and configure proactive log management.

CentOS Stream / Rocky Linux: Cleanup and Manage Massive systemd Journald Log Files

Introduction

As an experienced system administrator, you've likely encountered the insidious problem of disk space mysteriously dwindling on production servers. A frequent culprit, especially on modern Linux distributions utilizing systemd, is the journald logging service. While journald offers a robust, centralized logging solution, its default configurations can sometimes lead to log files consuming vast amounts of storage, silently pushing your system towards critical "disk full" states. This guide provides a highly technical, step-by-step approach to identify, clean up, and proactively manage systemd journal logs on CentOS Stream and Rocky Linux environments, ensuring system stability and optimal disk utilization.

Symptom & Error Signature

The primary symptom is a sudden or gradual reduction in available disk space, often leading to performance degradation, application failures, and even system instability. While journald itself doesn't typically produce "error signatures" indicating its own bloat, the consequences of its excessive disk usage manifest in various ways:

1. Low Disk Space Alerts: Monitoring systems will trigger alerts for partitions, especially / or /var, reaching critical thresholds.
2. System Slowness/Unresponsiveness: Applications may fail to write temporary files or logs, leading to errors or freezes.
3. Application Failures: Services might fail to start or operate correctly due to inability to allocate resources or write log entries.
4. Inability to Create New Files: When disk is completely full, even simple commands like touch will fail with "No space left on device".

You might observe output similar to this when checking disk usage:

df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G     0   16G   0% /dev/shm
tmpfs           6.3G  9.1M  6.3G   1% /run
/dev/sda2        50G   48G  2.0G  96% /  # <--- Critical / partition usage
tmpfs            16G  4.0K   16G   1% /tmp
/dev/sda1       976M  258M  651M  29% /boot
tmpfs           3.2G     0  3.2G   0% /run/user/0

A quick check of journalctl's current disk usage might reveal the culprit:

journalctl --disk-usage
Archived and active journals take up 45.0G in /var/log/journal.

Or a directory listing:

du -sh /var/log/journal/
45G     /var/log/journal/

Root Cause Analysis

The systemd-journald service is responsible for collecting and storing log data from various sources (kernel, system services, applications). By default, on CentOS Stream and Rocky Linux, journald stores logs persistently in /var/log/journal/. Without explicit configuration, journald's default behavior can lead to unbounded growth for several reasons:

1. Default Persistence: Logs are saved persistently across reboots by default, leading to continuous accumulation.
2. Lack of Explicit Size Limits: While journald has internal heuristics to prevent consuming all disk space, its default limits (SystemMaxUse=10% of file system size, SystemKeepFree=15% of file system size) are often too generous for dedicated logging or root partitions on large disks, or insufficient to handle high log volumes.
3. Verbose Application Logging: Misconfigured applications or services, especially those in debugging mode, can generate an enormous volume of log messages.
4. System Instability/Errors: A flapping service or a system experiencing frequent errors can flood the journal with repetitive error messages.
5. Noisy Kernel/Hardware: Certain kernel modules or hardware issues can cause continuous logging, such as I/O errors, network events, or hardware sensor warnings.

The core issue is that while systemd aims for convenience, its default log retention policies are often not optimized for environments with limited disk space, high log velocity, or specific compliance requirements.

Step-by-Step Resolution

This section will guide you through identifying journald's current usage, immediately reclaiming disk space, and then configuring permanent limits to prevent future occurrences.

#### 1. Assess Current Journald Disk Usage

Before making any changes, confirm that journald is indeed the primary consumer of your disk space.

journalctl --disk-usage

This command will display the total size of all active and archived journal files. If this value is significant (e.g., several GBs or tens of GBs), proceed with cleanup.

#### 2. Perform Immediate Log Cleanup (Temporary)

You can clean up old journal entries using journalctl's vacuum options. This is a temporary measure to free up space now.

a. Clean by Size

Remove old journal files until the total disk usage falls below a specified size. This is useful for immediately reclaiming a large chunk of disk space.

sudo journalctl --vacuum-size=1G

This command will delete archived journal files until the total disk usage is below 1GB. Adjust 1G to a size appropriate for your server (e.g., 500M, 2G).

b. Clean by Time

Remove journal files older than a specified duration. This is effective if you only need to retain logs for a certain period.

sudo journalctl --vacuum-time=7d

This command will delete all journal entries older than 7 days. Common time units include s, m, h, d, w, M (for month), y (for year).

The vacuum commands only delete archived journal files, not the currently active ones. To free up maximum space immediately, you might need to combine vacuum-size and vacuum-time if the initial cleanup isn't sufficient. journalctl --vacuum-size=1G --vacuum-time=7d will enforce both limits.

#### 3. Configure Persistent Journald Log Limits

To prevent journald logs from growing uncontrollably again, you need to modify its configuration file.

a. Edit the journald.conf Configuration File

The primary configuration file for journald is located at /etc/systemd/journald.conf.

sudo vi /etc/systemd/journald.conf

Locate and uncomment (remove the # symbol) or add the following lines, setting values appropriate for your environment.

[Journal]
#Storage=persistent
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=1000
SystemMaxUse=500M
SystemKeepFree=100M
#SystemMaxFileSize=
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emergency
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/tty12
#Audit=yes
#FlushIntervalSec=120s
#LogLevel=info

Here's an explanation of the most important parameters for log size management:

• SystemMaxUse: The maximum total disk space that the persistent journal files (in /var/log/journal) may occupy. When the disk space occupied by journal files grows beyond this limit, the oldest files are deleted until the usage is back under the limit. Recommended value: 500M to 2G for most servers.
• SystemKeepFree: Specifies the minimum amount of disk space to be kept free on the file system storing the persistent journal files. journald will clean up older files if the free space falls below this. This prevents journald from filling up the entire disk. Recommended value: 100M to 500M.
• SystemMaxFileSize: The maximum individual size of a journal file. Once a file reaches this size, it will be rotated. This helps manage individual file sizes for easier inspection. Recommended value: 50M to 200M.
• RuntimeMaxUse: Similar to SystemMaxUse, but for volatile journal files stored in /run/log/journal (which are cleared on reboot).
• RuntimeKeepFree: Similar to SystemKeepFree, but for volatile journal files.

b. Example Configuration

For a typical web server with moderate logging, a good starting point for /etc/systemd/journald.conf would be:

[Journal]
Storage=persistent
SystemMaxUse=1G
SystemKeepFree=200M
SystemMaxFileSize=100M

Setting SystemMaxUse or SystemKeepFree too low might lead to very short log retention, potentially hindering troubleshooting efforts. Always balance disk space savings with diagnostic needs. For production systems, retaining at least 7-14 days of logs is often advisable, adjusting SystemMaxUse accordingly.

c. Restart systemd-journald Service

After modifying journald.conf, you must restart the systemd-journald service for the changes to take effect.

sudo systemctl restart systemd-journald

The systemd-journald service can usually be restarted without affecting other running services, as systemd buffers log messages internally. However, it's always best practice to perform such actions during a maintenance window or when you can monitor for any unexpected behavior.

#### 4. Verify New Configuration and Monitor

After restarting the service, verify that the new settings are being honored.

journalctl --disk-usage

It might take some time for the old journal files to be vacuumed according to the new SystemMaxUse and SystemKeepFree limits, especially if a large amount of data needs to be processed. journald cleans up periodically.

You can also check the configuration directly:

systemctl cat systemd-journald.service | grep ExecStart

This shows the startup command, but to truly see the applied configuration, you'd typically examine /etc/systemd/journald.conf. The journalctl --disk-usage command is the most direct way to observe the result of the configuration.

#### 5. Identify Noisy Services (Optional, Advanced Troubleshooting)

If your journal fills up quickly even with strict limits, it indicates a "noisy" service. Identifying this service can help you address the root cause by adjusting its logging level or fixing underlying issues.

a. Monitor Live Log Output

sudo journalctl -f

This command displays new log entries in real-time. Observe which services are generating a high volume of messages.

b. Filter Logs by Service Unit

If you suspect a particular service, you can filter its logs:

sudo journalctl -u <service-name> -f

Replace <service-name> with the actual systemd unit name (e.g., nginx, php-fpm, mysqld).

c. Adjust Service Logging Level

If a specific application or service is excessively verbose, you can often configure its logging level within its own configuration file (e.g., nginx.conf, php-fpm.conf, my.cnf). For systemd services, you can also override the StandardOutput or StandardError settings, or direct output to a specific file instead of the journal, though this often deviates from journald's centralized logging benefits.

For example, to change the log level for a service that logs to stderr, you might create an override file:

sudo systemctl edit <service-name>

This opens an editor. Add content like:

[Service]
LogLevel=warning  # Or debug, info, notice, warning, error, crit, alert, emerg

Then save and exit. Note that LogLevel is a systemd general option, not specific to journald filtering from services. For application-specific logging control, it's usually done within the application's own configuration.

Modifying logging levels for critical services should be done carefully. Reducing log verbosity too much might make future troubleshooting difficult. Only do this if you understand the implications and have other monitoring in place.