Troubleshooting PostgreSQL shared_buffers Memory Allocation Crash on Ubuntu 20.04 LTS

Resolve critical PostgreSQL startup failures on Ubuntu 20.04 due to misconfigured shared_buffers exceeding system or kernel shared memory limits.

Resolve critical PostgreSQL startup failures on Ubuntu 20.04 due to misconfigured shared_buffers exceeding system or kernel shared memory limits.

When managing high-performance PostgreSQL databases, optimizing memory usage through parameters like shared_buffers is crucial. However, misconfiguring this setting can lead to a critical PostgreSQL service crash upon startup, presenting as a "Cannot allocate memory" error specifically related to shared memory segments. This guide will walk you through diagnosing and resolving this issue on Ubuntu 20.04 LTS systems.

Symptom & Error Signature

The most common symptom is that your PostgreSQL service fails to start, leading to applications being unable to connect to the database. When checking the service status or logs, you'll encounter FATAL errors indicating a failure to allocate shared memory.

You can observe this by checking your PostgreSQL logs or journalctl:

sudo systemctl status postgresql

● postgresql.service - PostgreSQL RDBMS
     Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2023-10-27 10:00:05 UTC; 5s ago
    Process: 12345 ExecStart=/usr/lib/postgresql/12/bin/pg_ctlcluster --skip-systemctl-redirect 12 main start (code=exited, status=1/FAILURE)
   Main PID: 12345 (code=exited, status=1/FAILURE)

Oct 27 10:00:05 webserver systemd[1]: Starting PostgreSQL RDBMS...
Oct 27 10:00:05 webserver pg_ctlcluster[12345]: FATAL:  could not create shared memory segment: Cannot allocate memory
Oct 27 10:00:05 webserver pg_ctlcluster[12345]: DETAIL:  Failed system call was shmget(key=5432001, size=16777216000, 0x1FF).
Oct 27 10:00:05 webserver pg_ctlcluster[12345]: HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded available system memory or kernel limits.
Oct 27 10:00:05 webserver pg_ctlcluster[12345]: The PostgreSQL process would require 16777216000 bytes of shared memory.
Oct 27 10:00:05 webserver pg_ctlcluster[12345]: Check for other PostgreSQL instances running. If not, try reducing the request size (e.g., by reducing shared_buffers or max_connections), or check system limitations.
Oct 27 10:00:05 webserver pg_ctlcluster[12345]: LOG:  database system is shut down
Oct 27 10:00:05 webserver systemd[1]: postgresql.service: Main process exited, code=exited, status=1/FAILURE
Oct 27 10:00:05 webserver systemd[1]: postgresql.service: Failed with result 'exit-code'.
Oct 27 10:00:05 webserver systemd[1]: Failed to start PostgreSQL RDBMS.

Or by examining the PostgreSQL log file directly (e.g., for PostgreSQL 12):

sudo tail -n 50 /var/log/postgresql/postgresql-12-main.log

2023-10-27 10:00:05.123 UTC [12345] LOG:  starting PostgreSQL 12.16 (Ubuntu 12.16-0ubuntu0.20.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0, 64-bit
2023-10-27 10:00:05.123 UTC [12345] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2023-10-27 10:00:05.123 UTC [12345] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-10-27 10:00:05.124 UTC [12345] FATAL:  could not create shared memory segment: Cannot allocate memory
2023-10-27 10:00:05.124 UTC [12345] DETAIL:  Failed system call was shmget(key=5432001, size=16777216000, 0x1FF).
2023-10-27 10:00:05.124 UTC [12345] HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded available system memory or kernel limits.  The PostgreSQL process would require 16777216000 bytes of shared memory.  Check for other PostgreSQL instances running. If not, try reducing the request size (e.g., by reducing shared_buffers or max_connections), or check system limitations.
2023-10-27 10:00:05.124 UTC [12345] LOG:  database system is shut down

Root Cause Analysis

The FATAL: could not create shared memory segment: Cannot allocate memory error, specifically with the shmget system call failure, indicates that PostgreSQL attempted to allocate a shared memory segment that either exceeded the physical memory available on the system or, more commonly, violated the Linux kernel's System V IPC (Interprocess Communication) shared memory limits.

PostgreSQL uses a significant portion of its memory for shared_buffers, which acts as a cache for database pages. This memory is allocated from the kernel's shared memory pool. The Linux kernel has two primary parameters that govern System V shared memory:

  1. kernel.shmmax: This parameter defines the maximum size (in bytes) of a single shared memory segment that the kernel will allow. If PostgreSQL's shared_buffers (which is typically one large shared memory segment) configured value exceeds kernel.shmmax, the allocation will fail.
  2. kernel.shmall: This parameter defines the total amount of shared memory (in pages) that the system can use at once. If the sum of all shared memory segments requested by all processes (including PostgreSQL) exceeds kernel.shmall multiplied by the system's PAGE_SIZE, allocation will fail. On x86-64 systems, PAGE_SIZE is typically 4096 bytes (4KB).

The crash typically occurs when:

  • You've set shared_buffers in postgresql.conf to a value too high for your system's physical RAM.
  • The shared_buffers value, even if reasonable for RAM, exceeds the current kernel.shmmax limit.
  • The overall shared memory footprint requested (primarily shared_buffers) exceeds the total allowed by kernel.shmall.

Step-by-Step Resolution

Follow these steps to diagnose and resolve the PostgreSQL shared buffers memory allocation crash.

1. Verify PostgreSQL Service Status and Examine Logs

First, confirm that the PostgreSQL service is indeed failing and check the logs for the exact error message.

# Check service status
sudo systemctl status postgresql

# View recent PostgreSQL specific logs (PostgreSQL 12)
sudo journalctl -u postgresql --no-pager | grep -i "fatal"

# Alternatively, check the main PostgreSQL log file
sudo tail -n 100 /var/log/postgresql/postgresql-12-main.log

Confirm the presence of the FATAL: could not create shared memory segment: Cannot allocate memory error. Note the size mentioned in the DETAIL line, as this is the amount PostgreSQL tried to allocate.

2. Identify Current shared_buffers Configuration

Locate your postgresql.conf file and check the configured shared_buffers value. For PostgreSQL 12 on Ubuntu 20.04, it's typically located at /etc/postgresql/12/main/postgresql.conf.

# Display the shared_buffers line from your config
grep -E '^shared_buffers' /etc/postgresql/12/main/postgresql.conf

3. Determine Current Kernel Shared Memory Limits

Check the current values of kernel.shmmax and kernel.shmall on your system.

# View current kernel shared memory maximum segment size
sudo sysctl kernel.shmmax

# View current kernel shared memory total pages
sudo sysctl kernel.shmall

# Get system page size (usually 4096 bytes or 4KB)
getconf PAGESIZE

The HINT in the PostgreSQL error message will often provide the exact byte size that PostgreSQL attempted to allocate. This is your target value for kernel.shmmax. For kernel.shmall, you'll need to divide that byte size by your system's PAGE_SIZE (e.g., 4096).

4. Option A: Reduce shared_buffers (Recommended First Approach)

If your shared_buffers value is excessively high (e.g., more than 25-30% of your total system RAM on a dedicated DB server, or even less if running other services), reducing it is often the simplest and safest fix.

  1. Edit postgresql.conf:

    sudo nano /etc/postgresql/12/main/postgresql.conf

  2. Adjust shared_buffers: Find the shared_buffers line and set it to a more conservative value. A common recommendation is 25% of your system's total RAM. For example, if you have 8GB of RAM, 2GB would be a good starting point. If you have 4GB RAM, 1GB is appropriate.

    # Example: Reduce shared_buffers to 2GB
shared_buffers = 2GB

    Setting shared_buffers too high can lead to excessive swapping, degrading performance, or causing system instability even if the kernel limits are met. Always ensure your shared_buffers value is well within your available physical RAM.

  3. Save and Exit: Press Ctrl+X, then Y to save, and Enter to confirm the filename.

  4. Restart PostgreSQL:

    sudo systemctl restart postgresql

  5. Verify Status:

    sudo systemctl status postgresql

    If successful, the service should now be active (running).

5. Option B: Increase Kernel Shared Memory Limits (If Reduction is Not Desirable)

If you have ample physical RAM and genuinely require a larger shared_buffers value (e.g., 4GB or more on systems with 16GB+ RAM), you can increase the kernel's shared memory limits. This requires calculating appropriate shmmax and shmall values.

  1. Determine Required Values: Let's assume you want to set shared_buffers = 4GB.

    • kernel.shmmax: This should be at least the value of shared_buffers in bytes. 4 GB = 4 * 1024 * 1024 * 1024 = 4294967296 bytes So, kernel.shmmax = 4294967296

    • kernel.shmall: This is calculated as the desired total shared memory in bytes, divided by the system's page size, then rounded up. Assuming PAGE_SIZE = 4096 bytes: 4294967296 bytes / 4096 bytes/page = 1048576 pages So, kernel.shmall = 1048576

  2. Create/Edit a sysctl.d Configuration File: It's best practice to create a new .conf file in /etc/sysctl.d/ rather than modifying /etc/sysctl.conf directly.

    sudo nano /etc/sysctl.d/99-postgresql-shm.conf

  3. Add/Modify Parameters: Add the calculated values to the file.

    # PostgreSQL shared memory settings
kernel.shmmax = 4294967296
kernel.shmall = 1048576

  4. Save and Exit: Press Ctrl+X, then Y to save, and Enter to confirm the filename.

  5. Apply New Kernel Parameters:

    sudo sysctl -p /etc/sysctl.d/99-postgresql-shm.conf

    You should see output confirming the new values:

    kernel.shmmax = 4294967296
kernel.shmall = 1048576

    These changes are persistent across reboots due to being in /etc/sysctl.d/.

  6. Verify New Kernel Parameters:

    sudo sysctl kernel.shmmax kernel.shmall

  7. Adjust shared_buffers in postgresql.conf (if not already done): If you increased kernel limits to support a larger shared_buffers value, ensure postgresql.conf reflects this.

    sudo nano /etc/postgresql/12/main/postgresql.conf

    Set shared_buffers to your desired value (e.g., 4GB).

  8. Restart PostgreSQL:

    sudo systemctl restart postgresql

  9. Verify Status:

    sudo systemctl status postgresql

    The service should now be active (running).

Always ensure that the sum of shared_buffers and other memory-intensive PostgreSQL settings (like work_mem, maintenance_work_mem, wal_buffers, and memory for max_connections) does not exceed your system's physical RAM to prevent excessive swapping and performance degradation. As a general rule for shared_buffers, start with 25% of system RAM and never exceed 50% without very specific reasons and careful monitoring.