Troubleshooting PostgreSQL: Disk Space Exhaustion (pg_log) and Lockfiles on Ubuntu 22.04 LTS

Resolve critical PostgreSQL database issues caused by full disk due to excessive pg_log files and potential stale postmaster.pid lockfiles on Ubuntu 22.04 LTS.

A PostgreSQL database service experiencing disk space exhaustion, particularly within its pg_log directory, can lead to severe operational disruptions. You might observe your applications failing to connect to the database, transactions timing out, or the PostgreSQL service itself being unable to start or crashing unexpectedly. This guide provides a highly technical, step-by-step resolution for addressing disk space issues primarily caused by unmanaged PostgreSQL log files and associated postmaster.pid lockfile complications on an Ubuntu 22.04 LTS system.

Symptom & Error Signature

When your PostgreSQL instance runs out of disk space, applications depending on it will typically report connection errors or transaction failures. On the server, you'll observe the PostgreSQL service in a failed state, often accompanied by "No space left on device" errors in system logs.

Typical Application Error (e.g., Ruby on Rails with PG gem):

ActiveRecord::StatementInvalid (PG::ConnectionBad: could not connect to server: Connection refused
    Is the server running on host "localhost" (127.0.0.1) and accepting
    TCP/IP connections on port 5432?)

Systemd Journal Log (sudo journalctl -xeu postgresql.service):

Aug 04 10:00:01 hostname postgresql[1234]: FATAL:  could not write to file "base/12345/12345": No space left on device
Aug 04 10:00:01 hostname postgresql[1234]: LOG:  terminating any other active server processes
Aug 04 10:00:01 hostname postgresql[1234]: WARNING:  terminating connection because of crash of another server process
Aug 04 10:00:01 hostname systemd[1]: postgresql.service: Main process exited, code=exited, status=1/FAILURE
Aug 04 10:00:01 hostname systemd[1]: postgresql.service: Failed with result 'exit-code'.
Aug 04 10:00:01 hostname systemd[1]: Failed to start PostgreSQL RDBMS.

PostgreSQL Log File (/var/log/postgresql/postgresql-*.log):

2026-08-04 10:00:01 UTC [1234]: [1-1] user=,db=,app=,client= FATAL:  could not write to file "base/12345/12345": No space left on device
2026-08-04 10:00:01 UTC [1234]: [2-1] user=,db=,app=,client= LOG:  terminating any other active server processes
2026-08-04 10:00:01 UTC [1234]: [3-1] user=app_user,db=mydb,app=[unknown],client=127.0.0.1 WARNING:  terminating connection because of crash of another server process
2026-08-04 10:00:01 UTC [1234]: [4-1] user=,db=,app=,client= LOG:  database system is shut down

Disk Space Check (df -h):

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       100G  100G     0 100% /

Root Cause Analysis

The core issue is a complete exhaustion of available disk space on the filesystem where PostgreSQL stores its data and/or logs. Specifically, the prompt highlights pg_log as the primary culprit, indicating that PostgreSQL's logging mechanism has generated an excessive volume of log files that have gone unchecked.

The chain of events typically unfolds as follows:

1. Unmanaged Log Growth: PostgreSQL, especially under heavy load or with verbose logging enabled, generates log entries rapidly. Without proper log rotation (e.g., via logrotate), these files accumulate indefinitely, eventually consuming all available disk space.
2. Disk Space Exhaustion: When the filesystem reaches 100% utilization, PostgreSQL can no longer write any data, including new log entries, temporary files, or even extend existing database files. This immediately halts database operations.
3. Database Crash/Failure to Start: Inability to write vital data or logs causes PostgreSQL processes to crash or fail during startup.
4. Stale postmaster.pid Lockfile: If PostgreSQL crashes mid-operation or fails to shut down gracefully due to the full disk, it may leave behind a postmaster.pid file in its data directory (e.g., /var/lib/postgresql/14/main/). This PID file acts as a lock, preventing another instance of PostgreSQL from starting, as it signifies that an instance is already running. Although a secondary symptom, it becomes an obstacle to restarting the database once space is freed.

While pg_log is explicitly mentioned, other potential contributors to disk space exhaustion could be large temporary files, uncleaned old backups, or an unexpectedly large pg_wal (Write-Ahead Log) directory if WAL archiving/streaming is misconfigured or stalled.

Step-by-Step Resolution

Follow these steps carefully to resolve the disk space issue and restore your PostgreSQL service.

1. Verify Disk Space Exhaustion and Identify Culprit

First, confirm that disk space is indeed the problem and pinpoint the directory consuming the most space.

# Check overall disk usage
df -h

# Check space used by PostgreSQL logs
sudo du -sh /var/log/postgresql

# Check space used by the entire PostgreSQL data directory
sudo du -sh /var/lib/postgresql

# Check overall log directory usage (might contain other large logs)
sudo du -sh /var/log

# Find the 20 largest files on the root filesystem (slow, use if previous checks aren't conclusive)
sudo find / -xdev -type f -size +1G -print0 | xargs -0 du -h | sort -rh | head -n 20

Identify any directory or file consuming a disproportionate amount of space, especially /var/log/postgresql/.

2. Create Emergency Free Space

You need to free up enough space for PostgreSQL to start and operate. Focus on removing old log files.

Exercise extreme caution when deleting files, especially with rm -rf. Double-check your commands and the target directory to avoid accidental data loss. Never delete files directly from /var/lib/postgresql/data/ (e.g., base/, pg_wal/) unless you are absolutely certain what you are doing and have a recent, verified backup.

# List PostgreSQL log files by size to identify the largest ones
ls -lh /var/log/postgresql

# Delete PostgreSQL log files older than 7 days
# Adjust '+7' as needed; for critical situations, you might delete all but the very newest logs.
sudo find /var/log/postgresql -name 'postgresql-*.log' -mtime +7 -delete

# If space is still critical and you need to keep recent logs, you can truncate large log files.
# This empties the file without deleting it. Be careful, this destroys content.
# Only use if `find -delete` didn't free enough space.
# Example: sudo truncate -s 0 /var/log/postgresql/postgresql-14-main.log

# Clear apt package cache (can sometimes be significant)
sudo apt clean

# Clear old systemd journal logs (can also be large)
sudo journalctl --vacuum-size=500M # Retain last 500MB of journal logs
sudo journalctl --vacuum-time=7d   # Retain last 7 days of journal logs

After performing these steps, re-check df -h to ensure sufficient space has been reclaimed (aim for at least 1-2GB free for critical operations).

3. Check PostgreSQL Service Status and Stale Lockfile

A full disk can cause PostgreSQL to crash, leaving a stale postmaster.pid file that prevents subsequent restarts.

# Check the current status of the PostgreSQL service
sudo systemctl status postgresql

If the service is failed, inactive, or stopping, proceed to check for a stale PID file.

# Locate the postmaster.pid file (adjust '14' for your PostgreSQL version)
sudo ls /var/lib/postgresql/14/main/postmaster.pid

If postmaster.pid exists, you must confirm that no PostgreSQL processes are actually running before deleting it. Deleting this file while PostgreSQL is active (even if systemctl reports otherwise due to a crash state) can lead to severe data corruption.

# Verify no PostgreSQL processes are running
ps aux | grep -E '^postgres.*postmaster'

If the ps aux command returns no output (other than the grep process itself), and systemctl status postgresql confirms the service is not running, it is safe to remove the stale PID file:

# Remove the stale PID file (adjust '14' for your PostgreSQL version)
sudo rm /var/lib/postgresql/14/main/postmaster.pid

4. Restart PostgreSQL

With disk space freed and any stale lockfiles removed, attempt to restart PostgreSQL.

# Start the PostgreSQL service
sudo systemctl start postgresql

# Check its status
sudo systemctl status postgresql

# Monitor logs for any new errors or successful startup messages
sudo tail -f /var/log/postgresql/postgresql-*.log
sudo journalctl -xeu postgresql.service -n 50

If PostgreSQL starts successfully, your applications should regain connectivity.

5. Implement Proactive Log Management

To prevent recurrence, configure logrotate for PostgreSQL logs. This is crucial for maintaining disk health.

# Open the logrotate configuration for PostgreSQL (path may vary slightly)
sudo nano /etc/logrotate.d/postgresql-common

Ensure the configuration is robust. A good logrotate configuration for PostgreSQL logs might look like this:

/var/log/postgresql/*.log {
    daily               # Rotate logs daily
    rotate 7            # Keep 7 old log files
    compress            # Compress rotated logs
    delaycompress       # Delay compression until the next rotation cycle
    missingok           # Don't error if log file is missing
    notifempty          # Don't rotate empty logs
    create 0640 postgres adm # Create new log file with specific permissions
    postrotate
        # Reload PostgreSQL to signal it to reopen log files.
        # This prevents logs from being written to the old, renamed file.
        # Check if the primary PostgreSQL socket exists before attempting to reload.
        if [ -e "/run/postgresql/.s.PGSQL.5432" ]; then
            pg_ctlcluster 14 main reload > /dev/null || true
        fi
    endscript
}

The pg_ctlcluster 14 main reload command assumes PostgreSQL version 14 and the default main cluster. Adjust 14 if your PostgreSQL version is different (e.g., 15, 16). The if condition ensures the command only runs if the PostgreSQL service is actively running.

Test the logrotate configuration:

# Perform a dry run to check for syntax errors
sudo logrotate -d /etc/logrotate.d/postgresql-common

# Force a rotation (use only after a successful dry run and once service is stable)
# This will immediately rotate logs based on the configuration.
sudo logrotate -f /etc/logrotate.d/postgresql-common

6. Implement Disk Space Monitoring

Proactive monitoring is your last line of defense. Integrate disk space monitoring into your existing infrastructure.

• Prometheus/Grafana: Deploy Node Exporter to collect disk usage metrics and configure Grafana dashboards with alerts.
• Zabbix/Nagios/Icinga: Configure checks (e.g., check_disk) to alert when disk usage exceeds a defined threshold (e.g., 80% or 90%).
• Simple Scripting: A cron job running a simple script to check df -h and email alerts can be a quick alternative.

Consider adjusting PostgreSQL's internal logging settings in /etc/postgresql/14/main/postgresql.conf as well. While logrotate handles file rotation, PostgreSQL settings control what gets logged. For instance, log_min_duration_statement can significantly increase log verbosity if set to 0 or a very low value. Review log_destination, logging_collector, log_filename, log_rotation_age, and log_rotation_size parameters. If log_rotation_age or log_rotation_size are enabled, they might conflict with logrotate; it's often best to let logrotate manage file rotation. After any changes, reload PostgreSQL: sudo systemctl reload postgresql.