Troubleshooting PostgreSQL shared_buffers Memory Limit Configuration Crash on CentOS Stream / Rocky Linux

Resolve PostgreSQL crashes on CentOS/Rocky Linux caused by insufficient shared_buffers memory limits. Learn to adjust kernel parameters and PostgreSQL configuration for stability.

Resolve PostgreSQL crashes on CentOS/Rocky Linux caused by insufficient shared_buffers memory limits. Learn to adjust kernel parameters and PostgreSQL configuration for stability.

PostgreSQL is a powerful object-relational database system, but like any complex software, it requires careful configuration, especially concerning memory management. A common issue encountered by administrators on CentOS Stream or Rocky Linux environments is PostgreSQL failing to start or crashing unexpectedly due to insufficient System V shared memory limits for its shared_buffers configuration. This guide will walk you through diagnosing and resolving this critical memory allocation error.

Symptom & Error Signature

When PostgreSQL encounters an issue allocating shared_buffers, your database service will typically fail to start or crash during operation. You'll observe applications failing to connect to the database, and the PostgreSQL service will show a "failed" status.

Typical error messages found in journalctl -u postgresql-<VERSION_NUMBER> or the PostgreSQL data directory logs (pg_log/):

FATAL: could not create shared memory segment: Cannot allocate memory
DETAIL: Failed system call was shmget(KEY, SIZE, 0600).
HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available RAM or Linux's system-wide shared memory limit. You might need to reduce the amount of shared memory requested by PostgreSQL, or reconfigure the kernel to allow more.  See Linux's shmmax and shmall parameters.

Another possible signature related to semaphores, though less common for this specific shared_buffers issue:

FATAL: could not create semaphores: No space left on device
DETAIL: Failed system call was semget(KEY, N, 0300).
HINT: This error does *not* mean you have run out of disk space.  It occurs when the OS has run out of System V semaphore sets, or values for semaphores.  You might need to increase the kernel's SHMMNI, SHMALL, and SEMMNS parameters.

If the system experiences severe memory pressure, you might also see messages from the Out-Of-Memory (OOM) Killer in dmesg or journalctl -k output:

kernel: Out of memory: Killed process 1234 (postgres) score 999
kernel: oom-kill:constraint=CC-GROUP,node=0,cpuset=/user.slice,mems_allowed=0,oom_score_adj=0

Root Cause Analysis

The core of this problem lies in how PostgreSQL allocates its main memory cache, shared_buffers, and how the Linux kernel manages System V shared memory segments.

  1. shared_buffers: This critical PostgreSQL parameter defines the amount of dedicated memory for caching data pages. A larger shared_buffers value generally improves database performance by reducing disk I/O. By default, PostgreSQL often suggests values around 25% of total system RAM, up to 8GB or more on systems with ample memory.

  2. System V Shared Memory: On Linux, shared_buffers typically uses System V shared memory segments (SHM). The kernel imposes limits on these segments to prevent a single process from monopolizing system resources or to ensure system stability. The two most relevant kernel parameters are:

    • kernel.shmmax: Defines the maximum size (in bytes) of a single shared memory segment that the kernel will allow. If your configured shared_buffers value in postgresql.conf exceeds this limit, PostgreSQL will fail to allocate it.
    • kernel.shmall: Defines the total amount of shared memory (in system pages) that can be allocated on the system at any given time. This is less frequently the direct cause of a crash for a single large shared_buffers allocation but can contribute if many processes are using shared memory or if shmmax is sufficiently large. A "page" size is typically 4096 bytes (4KB).

  3. CentOS/Rocky Linux Defaults: CentOS Stream and Rocky Linux often come with conservative default kernel.shmmax and kernel.shmall values. For instance, shmmax might be set to 18446744073692774399 (a very large number, practically infinite), but shmall might be limited. More commonly, if you're hitting issues, it's due to an shmmax that's too low for a significantly increased shared_buffers value, or simply not enough available RAM for the allocation.

  4. Allocation Failure: When PostgreSQL attempts to start, it requests a shared memory segment of the size specified by shared_buffers. If this request exceeds kernel.shmmax, or if there isn't enough physical memory available to back the segment (even if shmmax is high), the shmget() system call will fail, leading to the "Cannot allocate memory" error.

  5. OOM Killer: In rare cases, if PostgreSQL manages to start but the combined memory usage (including shared_buffers, work_mem, client connections, and other system processes) exceeds available RAM, the Linux Out-Of-Memory (OOM) Killer might terminate the PostgreSQL process to free up memory, leading to an unexpected crash.

Step-by-Step Resolution

Follow these steps to diagnose and correctly configure your system and PostgreSQL for stable operation.

1. Assess Current System Memory & PostgreSQL Configuration

First, gather information about your system's memory and current PostgreSQL settings.

# Check total system RAM
free -h

# Check current kernel shared memory parameters
sysctl -a | grep shm

# Find your PostgreSQL data directory and version (example for PG 14)
# You might need to adjust the service name based on your installation
sudo systemctl status postgresql-14

# The output will show something like:
# Active: active (running) since ...
# Docs:   https://www.postgresql.org/docs/14/static/
# Process: 1234 ExecStart=/usr/bin/postmaster ... -D /var/lib/pgsql/14/data (code=exited, status=0/SUCCESS)
# Main PID: 5678 (postmaster)

# Check your postgresql.conf for shared_buffers (replace with your actual data directory path)
# Example: If data directory is /var/lib/pgsql/14/data
grep shared_buffers /var/lib/pgsql/14/data/postgresql.conf

Note down the shared_buffers value you intend to use or currently have configured, and your system's total RAM.

2. Calculate Required Kernel Shared Memory Parameters

Based on your desired shared_buffers size, calculate the appropriate kernel.shmmax and kernel.shmall values.

The shared_buffers value should generally be between 15-25% of your total system RAM. Going much higher (e.g., 40%+) can be detrimental, as it can lead to double-buffering (OS page cache vs. PostgreSQL cache) and potentially less RAM for other processes. For very large RAM systems (64GB+), consider an upper bound of 8-16GB for shared_buffers and rely more on the OS page cache for remaining data.

Let's assume you want to set shared_buffers to 8GB (8192MB).

  • kernel.shmmax: This must be at least the size of your shared_buffers. It's generally safe to set it slightly higher or to a significant fraction of your total RAM.

    • Desired shared_buffers = 8 GB = 8 * 1024 * 1024 * 1024 bytes = 8589934592 bytes.
    • So, kernel.shmmax should be at least 8589934592. We can round up to a power of 2 or a convenient number, e.g., 8589934592 bytes.

  • kernel.shmall: This represents the total shared memory in pages. The default page size on x86-64 Linux is typically 4096 bytes (4KB).

    • Total shared memory (in bytes) = kernel.shmmax (or slightly more if you expect other SHM usage)
    • kernel.shmall = kernel.shmmax / PAGE_SIZE
    • kernel.shmall = 8589934592 / 4096 = 2097152

So, for an 8GB shared_buffers setting, you'd need: kernel.shmmax = 8589934592 kernel.shmall = 2097152

3. Adjust Kernel Parameters

To make these changes persistent across reboots, you should create a new sysctl configuration file.

# Create or edit a sysctl configuration file for PostgreSQL
sudo vi /etc/sysctl.d/99-postgresql.conf

Add the following lines to the file, replacing the values with your calculated ones:

# PostgreSQL shared_buffers configuration
kernel.shmmax = 8589934592
kernel.shmall = 2097152

While it's generally safe to set shmmax to a very large value (e.g., half of your RAM or more), ensure shmall is also appropriately set. Incorrectly high shmall with limited physical RAM can lead to system instability or the OOM Killer being invoked. Always ensure your total configured shared_buffers across all PostgreSQL instances (if more than one) plus other memory-consuming processes fit within your physical RAM.

Apply the changes immediately without rebooting:

sudo sysctl -p /etc/sysctl.d/99-postgresql.conf

Verify that the new values are active:

sysctl -a | grep shm

You should see output similar to:

kernel.shmmax = 8589934592
kernel.shmall = 2097152

4. Verify PostgreSQL shared_buffers Configuration

Now that the kernel can accommodate the larger shared memory segment, adjust your postgresql.conf if necessary.

# Open your postgresql.conf file (adjust path for your PostgreSQL version)
sudo vi /var/lib/pgsql/14/data/postgresql.conf

Find the shared_buffers line and ensure it's uncommented and set to your desired value (e.g., 8GB).

# shared_buffers = 128MB        # min 128kB
shared_buffers = 8GB            # Example: for an 8GB setting

While adjusting shared_buffers, also consider effective_cache_size. This parameter informs PostgreSQL about the total amount of memory available for caching (including the OS page cache). It should typically be set to 50-75% of your total RAM. This parameter does not allocate memory itself but influences the query planner. For example, if you have 16GB RAM and set shared_buffers = 4GB, you might set effective_cache_size = 12GB.

# effective_cache_size = 4GB
effective_cache_size = 12GB     # Example for 16GB RAM, 4GB shared_buffers

Save and close the postgresql.conf file.

5. Restart PostgreSQL Service

With kernel parameters and postgresql.conf updated, restart PostgreSQL.

# Replace '14' with your PostgreSQL version
sudo systemctl restart postgresql-14

Check the service status to ensure it started successfully:

sudo systemctl status postgresql-14

If it shows active (running), you're good to go. If not, carefully review the logs again (journalctl -u postgresql-14) for new errors.

6. Verify Post-Restart

Finally, connect to your PostgreSQL database and confirm the shared_buffers setting.

# Connect as the postgres user (or your superuser)
sudo -i -u postgres psql

# In the psql prompt, check the setting
SHOW shared_buffers;

The output should reflect your configured value:

 shared_buffers
----------------
 8GB
(1 row)

You have now successfully configured PostgreSQL's shared_buffers and the Linux kernel's shared memory parameters to prevent crashes due to memory allocation limits. Always monitor your system's memory usage to ensure optimal performance and stability.