PostgreSQL pg_log Disk Space Exhausted: Troubleshooting 'lock folder block' and Service Failure

Resolve critical PostgreSQL pg_log disk space exhaustion issues causing 'lock folder block' errors. Learn to free space, prevent recurrence, and restore database operations.

When a PostgreSQL database server runs out of disk space, especially due to an overgrown pg_log directory, it can lead to a cascade of critical failures. Your applications might become unresponsive, new database connections could fail, and PostgreSQL itself might refuse to start or operate correctly. The symptom “lock folder block” often arises from the system’s inability to write crucial lock files or other temporary data due to insufficient disk space, essentially blocking any further database operations. This guide provides a highly technical, accurate, and step-by-step resolution for this common and severe issue.

Symptom & Error Signature

The most prominent symptoms will be application errors indicating an inability to connect to the database, perform queries, or write data. On the server itself, you’ll observe PostgreSQL service failures and critical errors in its logs or system journals.

Typical error messages you might encounter include:

Application-level errors:

FATAL:  remaining connection slots are reserved for non-replication superuser connections
SQLSTATE[53100]: disk_full: ERROR:  could not write to file "base/pgsql_tmp/..." : No space left on device

PostgreSQL server logs (/var/lib/postgresql/1X/main/pg_log/ or journalctl -u postgresql):

LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
WARNING:  could not write to log file: No space left on device
FATAL:  could not write to file "pg_wal/archive_status/000000010000000000000001.00000000.backup": No space left on device
PANIC:  could not write to file "pg_xact/0000": No space left on device
LOG:  could not write to log file: No space left on device
FATAL:  the database system is shutting down

The “lock folder block” might not be a literal error message, but rather an observed behavior where the postmaster process fails to create or update its postmaster.pid lock file, or other processes fail to acquire locks because any write operation, including lock file management, is blocked by the full disk. This prevents the database from starting or running.

Root Cause Analysis

The primary root cause for “pg_log disk space exhausted” is straightforward: the /var/lib/postgresql/1X/main/pg_log directory has consumed all available disk space on its mounted partition, or the entire root filesystem. This happens due to:

  1. Excessive Logging: PostgreSQL’s logging level (log_min_messages, log_statement) might be set too verbosely (e.g., log_statement = 'all') for a production environment, generating an immense volume of log data.
  2. Lack of Log Rotation: The log_rotation_age or log_rotation_size parameters in postgresql.conf are not configured, or a system-level log rotation utility like logrotate is not properly set up for PostgreSQL logs. This allows log files to accumulate indefinitely.
  3. High Transaction Volume / Errors: A very busy database or an application generating frequent errors can rapidly fill logs, especially if combined with verbose logging.
  4. Unmonitored Disk Usage: Absence of proactive disk space monitoring and alerting systems allows the problem to escalate unnoticed until it causes a production outage.

When the disk becomes full, PostgreSQL cannot:

  • Write new log entries.
  • Create or update temporary files (e.g., for complex queries, sorts).
  • Perform WAL (Write-Ahead Log) archiving or even write new WAL segments.
  • Update critical system files, including the postmaster.pid lock file which signals the running status of the server.
  • Allocate memory or swap space effectively if the /tmp or swap partitions are also affected.

This inability to perform basic write operations leads to the database system becoming unstable, unresponsive, or failing to start altogether.

Step-by-Step Resolution

Addressing this issue requires immediate action to free up space, followed by preventive measures to ensure it doesn’t recur.

1. Immediate Action: Free Up Disk Space

The very first step is to free up enough disk space to allow PostgreSQL to start and function.

a. Identify the Full Filesystem: Use df -h to see disk usage across all mounted filesystems. This will confirm which partition is full and likely hosting your PostgreSQL data.

df -h

Look for a filesystem with 100% (or very close to it) Use%. It’s commonly / (root), /var, or a dedicated /var/lib/postgresql mount.

b. Locate Large Files/Directories: Navigate to the PostgreSQL data directory, usually /var/lib/postgresql/1X/main/, where 1X is your PostgreSQL version (e.g., 14). Then, examine the pg_log directory.

# Replace '14' with your PostgreSQL version
sudo su - postgres
cd /var/lib/postgresql/14/main/
du -sh pg_log/*
du -sh pg_log/
exit

This will show you the size of individual log files and the total size of the pg_log directory.

c. Safely Remove Old pg_log Files: This is the most critical immediate step. You need to delete old log files. Do NOT delete currently open log files or the entire pg_log directory.

[!WARNING] Be extremely careful when deleting files on a production system. Incorrect rm commands can lead to data loss. Always double-check your find and rm commands before execution. Only delete files you are certain are old log files and not actively being written to.

To delete log files older than, say, 7 days:

# Replace '14' with your PostgreSQL version
sudo find /var/lib/postgresql/14/main/pg_log -name "postgresql-*.log" -mtime +7 -exec rm {} \;
  • -name "postgresql-*.log": Targets PostgreSQL log files. Adjust if your naming convention differs.
  • -mtime +7: Selects files modified more than 7 days ago. Adjust as needed to free sufficient space.
  • -exec rm {} \;: Executes the rm command on each found file.

If you need more space immediately, consider deleting files older than 3 or 1 day, or even specific large files.

# Example: Delete files older than 3 days
sudo find /var/lib/postgresql/14/main/pg_log -name "postgresql-*.log" -mtime +3 -exec rm {} \;

# If you are absolutely desperate, you could delete all but the last few hours/days,
# but use with extreme caution:
# List files by modification time, exclude very recent ones, and pipe to rm
# sudo find /var/lib/postgresql/14/main/pg_log -name "postgresql-*.log" -mmin +60 -print0 | xargs -0 rm

d. Check Disk Space Again: After deleting files, run df -h again to confirm that space has been freed.

df -h

You should now see a lower Use% and more Avail space.

2. Restart PostgreSQL Service

Once sufficient disk space is available, attempt to restart PostgreSQL.

sudo systemctl start postgresql
sudo systemctl status postgresql

[!NOTE] If PostgreSQL still fails to start, check journalctl -u postgresql -xn 100 for recent errors and verify there’s enough free space for temporary files and WAL segments. Sometimes, after a disk full event, WAL consistency checks can take time.

3. Implement Log Rotation (Preventive)

To prevent future disk space exhaustion, configure logrotate for your PostgreSQL logs.

a. Create/Edit logrotate Configuration: Create a new file or edit an existing one in /etc/logrotate.d/ for PostgreSQL.

sudo nano /etc/logrotate.d/postgresql

Add the following configuration (adjust paths and numbers as per your setup):

# Replace '14' with your PostgreSQL version if it differs
/var/lib/postgresql/14/main/pg_log/*.log {
    daily
    missingok
    rotate 7
    compress
    delaycompress
    notifempty
    create 0640 postgres postgres
    sharedscripts
    postrotate
        # Reload PostgreSQL to make it open new log files.
        # This sends a SIGHUP signal to the postmaster.
        # Ensure this command is correct for your system/init system.
        if [ -f "/var/run/postgresql/.s.PGSQL.5432" ]; then
            pg_ctlcluster 14 main reload > /dev/null || true
        fi
    endscript
}
  • daily: Rotate logs daily. Can be weekly or monthly.
  • rotate 7: Keep 7 rotated log files.
  • compress, delaycompress: Compress old logs.
  • create 0640 postgres postgres: Create new log files with specified permissions and ownership.
  • postrotate: Commands to run after rotation. pg_ctlcluster 14 main reload (for Debian/Ubuntu) or systemctl reload postgresql sends a SIGHUP to PostgreSQL, making it close current log files and open new ones.

b. Test logrotate Configuration: You can manually run logrotate in debug mode to check the configuration.

sudo logrotate -d /etc/logrotate.d/postgresql

This will show what logrotate would do without actually performing the actions.

4. Optimize PostgreSQL Logging Configuration (Preventive)

Adjust postgresql.conf to control the volume and rotation of logs directly from PostgreSQL. This is supplementary to logrotate and can provide finer control.

a. Edit postgresql.conf: Locate your postgresql.conf file, typically at /etc/postgresql/1X/main/postgresql.conf.

# Replace '14' with your PostgreSQL version
sudo nano /etc/postgresql/14/main/postgresql.conf

b. Adjust Logging Parameters: Find and modify (or add) the following parameters. Uncomment them if they are commented out.

# --- FILE LOCATIONS ---
log_directory = 'pg_log'             # directory where log files are written, relative to PGDATA
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' # log file name pattern; use 'csvlog' for CSV format
log_rotation_age = 1d                # Automatic log file rotation will occur after this much time.
log_rotation_size = 0                # Automatic log file rotation will occur after this much data is written.
                                     # Set to 0 to use log_rotation_age only, or a value like '100MB'.
log_truncate_on_rotation = on        # If ON, PostgreSQL will overwrite existing log files with the same name.
                                     # This is useful with log_rotation_age when you don't use a time-based filename.
                                     # With the above log_filename, it's less critical.
logging_collector = on               # Enable saving of log output to log_directory

# --- WHAT TO LOG ---
log_min_messages = warning           # Minimum message level to log (DEBUG5, DEBUG4, DEBUG3, DEBUG2, DEBUG1, INFO, NOTICE, WARNING, ERROR, LOG, FATAL, PANIC)
                                     # Set to 'warning' or 'error' for production to reduce log volume.
log_min_error_statement = error      # Statements generating this level (or higher) of error will be logged.
log_statement = 'none'               # 'none', 'ddl', 'mod', 'all'
                                     # Set to 'none' in production unless debugging.
log_duration = off                   # Log the duration of each completed statement. Useful for performance tuning.

[!IMPORTANT] For log_rotation_age and log_rotation_size, if logrotate is configured to handle the files, you might set log_truncate_on_rotation = on and rely solely on logrotate for actual file deletion. If log_rotation_size is set to 0 and log_rotation_age to 1d, PostgreSQL will create a new log file every day, and logrotate will then manage rotating and compressing these daily files.

c. Reload PostgreSQL Configuration: After modifying postgresql.conf, you need to reload PostgreSQL for the changes to take effect. A reload is usually sufficient; a full restart is not always required unless logging_collector itself was changed from off to on.

sudo systemctl reload postgresql

5. Monitor Disk Usage (Long-term Prevention)

Implement robust monitoring to prevent recurrence.

  • Monitoring Tools: Use tools like Prometheus with node_exporter for disk metrics, Zabbix, Nagios, or Icinga to monitor disk space usage on your PostgreSQL server.
  • Alerting: Configure alerts to notify you (via email, Slack, PagerDuty, etc.) when disk usage exceeds a predefined threshold (e.g., 80% or 90%).

6. Consider WAL Archiving Configuration (Advanced)

If your setup uses WAL archiving for point-in-time recovery or replication, ensure it’s not contributing to disk space issues.

  • archive_mode and archive_command: Verify that archive_command is successfully moving WAL segments to their destination and that the destination itself has sufficient space and is being properly managed (e.g., old archives being deleted). Failed archive_command will cause WAL segments to accumulate in pg_wal/pg_archive or pg_wal (depending on PostgreSQL version), leading to disk exhaustion.

By following these steps, you can recover from a PostgreSQL disk space exhaustion incident and implement robust measures to prevent it from happening again, ensuring the stability and reliability of your database environment.