Troubleshooting PostgreSQL: pg_log Disk Space Exhausted & Database Lockfile Errors on Alpine Linux

Resolve critical PostgreSQL database lock and 'No space left' errors on Alpine Linux caused by full pg_log directories. A step-by-step guide.

When your PostgreSQL database running on Alpine Linux suddenly becomes inaccessible, applications fail to connect, or your website goes down, a common culprit is the database's pg_log directory consuming all available disk space. This critical condition often leads to PostgreSQL failing to start or operate, frequently presenting with "No space left on device" errors, and sometimes leaving behind stale lock files that further prevent startup. This guide provides a highly technical, step-by-step resolution for this debilitating issue.

Symptom & Error Signature

Users will typically observe applications failing to connect to the database, receiving connection refused errors, or seeing 5xx errors on their web applications. Attempts to restart or check the PostgreSQL service will reveal critical errors in logs.

Typical Log Output (from /var/lib/postgresql/data/pg_log/ or journalctl -u postgresql):

FATAL:  could not write to log file: No space left on device
LOG:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to terminate, because the postmaster itself is shutting down.
FATAL:  the database system is shutting down
LOG:  database system is shut down

PostgreSQL Service Status (rc-service postgresql status or systemctl status postgresql):

$ rc-service postgresql status
* status: crashed

# or if using systemd in a container
$ systemctl status postgresql
× postgresql.service - PostgreSQL RDBMS
     Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Mon 2026-08-16 10:30:00 UTC; 5min ago
    Process: 1234 ExecStart=/usr/bin/pg_ctl -D /var/lib/postgresql/data start (code=exited, status=1/FAILURE)
   Main PID: 1235 (code=exited, status=0/SUCCESS)
        CPU: 1.234s

[...]
Aug 16 10:30:00 server pg_ctl[1234]: pg_ctl: could not start server
Aug 16 10:30:00 server pg_ctl[1234]: Examine the log output.
Aug 16 10:30:00 server systemd[1]: postgresql.service: Control process exited, code=exited, status=1/FAILURE
Aug 16 10:30:00 server systemd[1]: postgresql.service: Failed with result 'exit-code'.

Potential dmesg or kernel log output:

[12345.678901] XFS (dm-0): filesystem is full, freespace is 0 B
[12345.678902] XFS (dm-0): write failed, error 28

Root Cause Analysis

The primary root cause for this issue is an exhausted disk space on the partition hosting the PostgreSQL data directory, specifically within the pg_log subdirectory. This typically occurs due to:

1. Verbose Logging: PostgreSQL configured to log extensively (e.g., log_statement = 'all', high log_min_duration_statement) without adequate log rotation.
2. Lack of Log Rotation: The pg_log directory continuously accumulates log files, eventually filling the disk. This is common in environments where system-level log rotation (like logrotate on Alpine) is not properly configured for PostgreSQL, or PostgreSQL's internal log rotation parameters are not set.
3. Long-Running Transactions/Errors: Persistent errors or extremely long-running transactions can generate an abnormal volume of log entries in a short period.
4. Small Root Partition: In containerized environments or virtual machines, the root partition (where /var/lib/postgresql/data often resides) might be provisioned with limited disk space, making it susceptible to filling up quickly.
5. Stale Lock Files: When PostgreSQL fails to shut down cleanly (e.g., due to a crash caused by full disk), it might leave behind a postmaster.pid file in the data directory. This file, containing the PID of the last running postmaster process, prevents subsequent startup attempts, falsely indicating that a PostgreSQL instance is already running. While often a symptom, it can become a blocking issue itself.

Step-by-Step Resolution

Follow these steps carefully to recover your PostgreSQL instance and prevent future occurrences.

1. Confirm Disk Space Exhaustion

First, verify that disk space is indeed the problem and identify the offending directory.

# Check overall disk usage
df -h

# Example output indicating a full partition (e.g., /dev/vda1 or /)
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        20G   20G     0 100% /
tmpfs           3.9G     0  3.9G   0% /dev/shm

Next, pinpoint the large files within the PostgreSQL data directory. The default data directory on Alpine for PostgreSQL is usually /var/lib/postgresql/data.

# Identify largest directories within the PostgreSQL data directory
# Ensure you replace /var/lib/postgresql/data with your actual data directory if different
du -sh /var/lib/postgresql/data/* | sort -rh | head -n 10

# Specifically check the pg_log directory
du -sh /var/lib/postgresql/data/pg_log

If du -sh /var/lib/postgresql/data/pg_log shows a significant percentage of the full disk, you've found your culprit.

2. Stop PostgreSQL Service

It's crucial to stop the PostgreSQL service before manipulating its data directory to prevent data corruption.

# On Alpine Linux using OpenRC (common in bare metal/VM installs)
rc-service postgresql stop

# If running in a container that uses systemd (less common for Alpine base image but possible)
# Or on other Linux distributions like Ubuntu/Debian
systemctl stop postgresql

Do NOT proceed with deleting or truncating files if PostgreSQL is still running. This can lead to severe data corruption.

3. Clear Log Files (Temporary Fix)

Now, free up some disk space by removing old log files.

# Navigate to the pg_log directory
cd /var/lib/postgresql/data/pg_log

# List log files by size (largest first) to identify targets for removal
ls -lh | sort -rh -k 5

# Option 1: Delete all log files older than N days (e.g., 7 days)
# Adjust the path to your pg_log directory if necessary
find /var/lib/postgresql/data/pg_log -type f -name "*.log" -mtime +7 -delete

# Option 2: Delete specific large log files (e.g., the oldest 5)
# BE CAREFUL: This deletes without confirmation. Review `ls -lh | sort -rh -k5` first.
ls -t *.log | tail -n 5 | xargs rm

# Option 3: Truncate current log files if they are still being held open
# This is less common but useful if a file is still growing and `rm` doesn't immediately free space.
# Find files currently held open by PostgreSQL (if any) and truncate them.
# Replace <PID> with the actual PostgreSQL process ID if it was running and created a huge log before crashing.
# For example, if 'du -sh' shows a huge log named 'postgresql-2026-08-16_000000.log'
# echo > postgresql-2026-08-16_000000.log

After clearing logs, verify disk space has been freed.

df -h

4. Address Stale Lock Files (postmaster.pid)

If PostgreSQL still fails to start after freeing disk space, a stale postmaster.pid lock file might be the culprit. This file prevents a new postmaster process from starting.

# Check for the existence of postmaster.pid
ls /var/lib/postgresql/data/postmaster.pid

# Check if any postgres processes are *actually* running (should not be, given step 2)
ps aux | grep -i postgres | grep -v grep

> [!IMPORTANT]
> ONLY remove `postmaster.pid` if you are absolutely certain that no PostgreSQL processes are running. Removing it while PostgreSQL is active can lead to severe data corruption and loss.

# If no postgres processes are running, safely remove the stale PID file
rm /var/lib/postgresql/data/postmaster.pid

# Check for other potential lock files (less common but good practice)
# e.g., /tmp/.s.PGSQL.5432.lock or similar, depending on your setup.

5. Start PostgreSQL Service

With disk space freed and potential lock files removed, attempt to start PostgreSQL.

# On Alpine Linux using OpenRC
rc-service postgresql start

# If using systemd
systemctl start postgresql

# Check status
rc-service postgresql status
# or
systemctl status postgresql

If it starts successfully, proceed to configure log rotation. If not, re-check logs for new errors.

6. Configure PostgreSQL Internal Log Rotation

Modify postgresql.conf to enable internal log rotation. This is the first line of defense against log file bloat.

# Open your postgresql.conf (path may vary, common locations: /var/lib/postgresql/data/postgresql.conf)
# On Alpine, it might be in /etc/postgresql/<version>/postgresql.conf or similar,
# with /var/lib/postgresql/data being a symlink or default for initdb.
vi /var/lib/postgresql/data/postgresql.conf

Find and adjust the following parameters:

# Logging Collector: Enables the logging collector (essential for internal log rotation)
logging_collector = on

# Log Destination: Where logs are written. 'stderr' writes to systemd/OpenRC logs, 'csvlog' or 'jsonlog' for structured.
# Using 'stderr' is often preferred for containerized environments to leverage stdout/stderr streams.
# If 'stderr' is used, ensure your container logging driver or host systemd journal is configured for rotation.
# If 'csvlog' or 'jsonlog' is used, logs go to pg_log and require direct rotation.
log_destination = 'stderr' # Or 'csvlog' or 'jsonlog'

# Log Directory: Where log files are stored when logging_collector is 'on' and not 'stderr'.
log_directory = 'pg_log'  # relative to PGDATA

# Log Filename: Format for log file names. '%Y-%m-%d_%H%M%S' is good for unique files.
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'

# Log Rotation Age: Rotate logs every 1 day
log_rotation_age = 1d

# Log Rotation Size: Rotate logs after 100MB (if age not reached)
log_rotation_size = 100MB

# Log Truncate on Rotation: If log_filename is not using time-based escapes,
# this truncates the existing log file instead of creating a new one.
# Generally, set to 'off' if using time-based filenames like above.
log_truncate_on_rotation = off

# Additional useful logging parameters (adjust as needed for diagnostics)
# log_min_duration_statement = 1000 # Log statements taking longer than 1 second
# log_connections = on
# log_disconnections = on
# log_lock_waits = on
# log_temp_files = 0 # Log temporary files larger than 0KB
# client_min_messages = warning # Set to debug for more verbose client messages
# log_line_prefix = '%m [%p] %q%u@%d ' # Example prefix for log lines

After modifying postgresql.conf, restart PostgreSQL for changes to take effect.

rc-service postgresql restart
# or
systemctl restart postgresql

7. Implement System-Level Log Rotation (logrotate for Alpine)

For robust log management, especially if you're writing logs directly to files (e.g., log_destination = 'csvlog'), configure logrotate. Alpine Linux uses logrotate by default.

# Ensure logrotate is installed
apk add logrotate

# Create or modify a logrotate configuration file for PostgreSQL
vi /etc/logrotate.d/postgresql

Add the following content to /etc/logrotate.d/postgresql. Adjust paths if your pg_log is elsewhere.

# /etc/logrotate.d/postgresql
/var/lib/postgresql/data/pg_log/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0640 postgres postgres
    sharedscripts
    postrotate
        # Signal PostgreSQL to reload its log file descriptors.
        # This prevents "file moved" errors and ensures new logs go to new files.
        # Check your PID file path.
        if [ -f /var/lib/postgresql/data/postmaster.pid ]; then
            kill -HUP $(cat /var/lib/postgresql/data/postmaster.pid)
        fi
    endscript
}

Explanation of logrotate directives:

• daily: Rotate logs daily.
• rotate 7: Keep 7 rotated log files.
• compress: Compress old log files using gzip.
• delaycompress: Compress the previous log file on the next rotation cycle. This is useful for applications that might still be writing to the log file being rotated.
• missingok: If the log file is missing, do not issue an error.
• notifempty: Do not rotate the log if it is empty.
• create 0640 postgres postgres: Create new log files with specific permissions and ownership after rotation.
• sharedscripts: Run postrotate script only once for all matched log files, not per file.
• postrotate/endscript: Commands to execute after rotation. kill -HUP $(cat /var/lib/postgresql/data/postmaster.pid) sends a HUP signal to the PostgreSQL postmaster process, instructing it to close its current log file and open a new one, preventing data loss during rotation.

Test logrotate (optional but recommended):

# Force logrotate to run for PostgreSQL config (dry run first)
logrotate -d /etc/logrotate.d/postgresql

# Force logrotate to run for PostgreSQL config (actual run)
logrotate -f /etc/logrotate.d/postgresql

Check the pg_log directory for new, rotated, and compressed files.

8. Monitor Disk Usage & Set Alarms

Finally, implement proactive monitoring to prevent this issue from recurring.

• Disk Usage Monitoring: Use tools like Prometheus, Nagios, Zabbix, or even simple shell scripts with df -h to monitor the disk space on the partition hosting your PostgreSQL data.
• Alerting: Configure alerts (email, Slack, PagerDuty) to trigger when disk usage exceeds a defined threshold (e.g., 80% or 90%).

By combining PostgreSQL's internal log rotation with system-level logrotate and proactive monitoring, you can ensure the stability and availability of your PostgreSQL databases on Alpine Linux.