Troubleshooting Docker Container Exit Code 137 (OOM Killed) on CentOS Stream / Rocky Linux

Resolve Docker containers unexpectedly terminating with exit code 137 due to Out-Of-Memory (OOM) killer on CentOS Stream and Rocky Linux systems.

Introduction

Encountering a Docker container that repeatedly exits with code 137 can be a frustrating and critical issue for web hosting environments, often indicating an Out-Of-Memory (OOM) event. When a container is OOM-killed, it means the Linux kernel's OOM killer has terminated the container's main process (or one of its child processes) because the system or the container itself has run out of available memory. This guide provides a highly technical, step-by-step approach to diagnose and resolve this issue specifically on CentOS Stream and Rocky Linux distributions.

Symptom & Error Signature

Users typically observe their Docker containers restarting unexpectedly, or failing to start completely. Checking the container's status via docker ps -a will reveal the "Exited (137)" status. Further investigation into the host system logs and container logs will confirm the OOM event.

Typical Error Output:

When checking stopped containers:

sudo docker ps -a

CONTAINER ID   IMAGE                 COMMAND                  CREATED         STATUS                         PORTS     NAMES
a1b2c3d4e5f6   my-app:latest         "nginx -g 'daemon of…"   2 minutes ago   Exited (137) 5 seconds ago             my-web-server

When inspecting container logs for the specific container:

sudo docker logs a1b2c3d4e5f6

(The container logs might not directly show the OOM event, as it's a kernel-level termination, but could show the application abruptly stopping.)

The definitive evidence of an OOM kill comes from the host system's kernel messages, accessible via dmesg or journalctl:

sudo dmesg | grep -i -E 'killed process|oom-kill|out of memory'

[12345.678901] Killed process 1234 (node) total-vm:4194304kB, anon-rss:2097152kB, file-rss:0kB, shmem-rss:0kB
[12345.678902] oom-kill:constraint=CONTAINER,node=0,sid=0,task_memcg=/docker/a1b2c3d4e5f6,task=node,pid=1234
[12345.678903] Memory cgroup out of memory: Killed process 1234 (node) total-vm:4194304kB, anon-rss:2097152kB, file-rss:0kB, shmem-rss:0kB

Or using journalctl:

sudo journalctl -k -p err | grep -i -E 'killed process|oom-kill|out of memory'

Root Cause Analysis

An Exited (137) status specifically indicates that the container received a SIGKILL signal (signal 9) from the Linux kernel. In almost all cases, this is initiated by the Out-Of-Memory (OOM) killer. The underlying reasons can be categorized as follows:

1. Container Memory Limit Exceeded: The most common cause. The Docker container was started with a finite memory limit (e.g., via --memory flag or memory in docker-compose.yml), and the application running inside the container attempted to consume more memory than allocated. The kernel's OOM killer, acting within the cgroup limits imposed by Docker, terminated the container's process to prevent it from exhausting its allocated resources.
2. Host System Memory Exhaustion: Even without specific Docker memory limits, if the entire host system runs out of physical RAM and swap, the OOM killer will step in to terminate processes, including Docker containers, to maintain system stability.
3. Application Memory Leak or Inefficiency: The application within the container might have a memory leak, steadily consuming more RAM over time until it hits the limit. Alternatively, the application might be poorly optimized, requiring significantly more memory for specific tasks than initially provisioned.
4. Incorrect Memory Estimation: The memory allocated to the container might be fundamentally underestimated for its workload, especially during peak usage or data processing spikes.
5. Swap Space Depletion: While Docker containers are typically advised to run without direct swap access (managed by the host), an overall lack of host swap space can exacerbate OOM situations if physical RAM is fully utilized.

The OOM killer's decision is based on an internal score (oom_score) for each process, favoring processes that consume a lot of memory and have low "importance."

Step-by-Step Resolution

Follow these steps to diagnose and resolve Docker OOM-killed containers on CentOS Stream / Rocky Linux.

1. Confirm OOM Event and Identify Affected Process

First, confirm that the exit code 137 is indeed due to an OOM event and identify which process was killed.

# Check the status of all containers, including exited ones
sudo docker ps -a

# Get the container ID of the exited container (e.g., 'a1b2c3d4e5f6')

# Check kernel messages for OOM events
# This is crucial for confirmation and identifying the killed process within the container
sudo dmesg | grep -i -E 'killed process|oom-kill|out of memory'

# For systemd-based logging, check journalctl for kernel messages
sudo journalctl -k -p err | grep -i -E 'killed process|oom-kill|out of memory'

Look for lines similar to Memory cgroup out of memory: Killed process <PID> (<PROCESS_NAME>). This confirms the OOM kill and points to the specific process (e.g., node, java, python) that was terminated.

2. Analyze Container Memory Usage and Limits

Determine if the container had explicit memory limits and how much memory it was actually using before termination.

# Get detailed information about the container, including configured memory limits
sudo docker inspect <container_id_or_name> | grep -E 'Memory|KernelMemory|Swap'

# Example output snippet:
# "Memory": 0,                    # 0 means no explicit limit
# "MemoryReservation": 0,
# "KernelMemory": 0,
# "Swap": 0,
# "Swappiness": null,
# "OomKillDisable": false,
# "MemorySwap": 0,                # If Memory is 0, this is also 0 (unlimited)
# "MemoryUsage": 1850123456,      # Current memory usage (bytes)
# "MaxMemoryUsage": 2097152000,   # Max memory usage observed (bytes)
# "Failcnt": 1,                   # Number of times memory allocation failed

# If the container is still running (or was briefly before being killed),
# you can monitor its live memory usage with `docker stats`.
# Run this before reproducing the error if possible, or on a similar container.
sudo docker stats <container_id_or_name> --no-stream

If Memory is 0 in docker inspect, it means the container had no explicit memory limit imposed by Docker. In this case, the OOM event was likely due to the host system running out of memory, or the container consuming excessive resources and being targeted by the host's OOM killer without Docker's cgroup isolation providing an earlier boundary.

3. Adjust Docker Container Memory Limits

Based on your analysis, the most direct solution is often to provide more memory to the container or optimize its usage.

A. Increase Memory for docker run:

If you're using docker run, add or increase the --memory flag. This sets the hard memory limit.

# Example: Allocate 4GB of RAM (4096m) to the container
sudo docker run -d --name my-web-server --memory="4096m" my-app:latest

# You can also set a soft limit (--memory-reservation) which attempts
# to keep the container below this, but allows bursting up to --memory.
sudo docker run -d --name my-web-server --memory="4096m" --memory-reservation="2048m" my-app:latest

B. Increase Memory for docker-compose.yml:

For docker-compose deployments, modify the deploy.resources.limits.memory setting in your docker-compose.yml file.

version: '3.8'
services:
  my-web-server:
    image: my-app:latest
    ports:
      - "80:80"
    deploy:
      resources:
        limits:
          memory: 4096m # Set hard memory limit to 4GB
          # Also consider memory_reservation for a soft limit
          # memory_reservation: 2048m

After modifying, restart your service:

sudo docker-compose up -d --build

Increment memory limits incrementally. Do not blindly allocate a very large amount of memory, as this can starve other services on the host or mask an underlying application memory leak. Always monitor usage after adjustment.

4. Optimize Application Memory Usage

If increasing limits doesn't resolve the issue or if the required memory becomes excessive, the problem might be within the application itself.

• Code Review and Profiling: Investigate the application code for potential memory leaks, inefficient data structures, or excessive caching. Use language-specific profiling tools (e.g., jemalloc for C/C++, pprof for Go, JVM Flight Recorder for Java, memory_profiler for Python) to identify memory hotspots.
• Garbage Collection Tuning: For runtimes like Java or Node.js, tune garbage collection parameters to be more aggressive or optimize heap sizes.
• Configuration Review: Check application configuration settings that might affect memory usage, such as cache sizes, connection pool limits, or buffer sizes.
• Update Dependencies: Outdated libraries or frameworks might have memory management bugs that have been fixed in newer versions.

5. Increase Host System Resources (If Host OOM)

If docker inspect showed no specific memory limits (Memory: 0), and dmesg indicated a general host-level OOM rather than a cgroup-specific one, then the host itself is running out of memory.

• Add Physical RAM: The most effective solution is to provision more physical RAM for your CentOS Stream / Rocky Linux server.

• Increase Swap Space: While not a direct replacement for RAM, increasing swap can provide a buffer.

  # Check current swap status
  sudo swapon --show
  sudo free -h
  
  # Create a new swap file (e.g., 8GB)
  sudo fallocate -l 8G /swapfile_new
  sudo chmod 600 /swapfile_new
  sudo mkswap /swapfile_new
  sudo swapon /swapfile_new
  
  # Make swap persistent across reboots by adding to /etc/fstab
  echo '/swapfile_new none swap sw 0 0' | sudo tee -a /etc/fstab
  
  # Adjust swappiness (optional, default is 30-60 on modern kernels)
  # Lower value means kernel tries to avoid swapping data out of RAM for as long as possible
  echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf
  sudo sysctl -p
  

  Relying heavily on swap can significantly degrade performance due to slower disk I/O compared to RAM. It should be considered a temporary solution or last resort for non-critical memory needs.

6. Configure OOM Score Adjustment for Critical Containers (Advanced)

For extremely critical services, you can influence the OOM killer's behavior using oom_score_adj. A negative value makes the process less likely to be killed, a positive value makes it more likely.

# Find the PID of the main process inside your running container
sudo docker top <container_id_or_name>

# Example output:
# PID    USER       TIME       COMMAND
# 1234   root       0:05       nginx -g daemon off;

# Change the OOM score adjustment for process 1234 on the host (less likely to be killed)
# Adjust the PID to match your container's main process.
# A value of -1000 makes it very unlikely to be killed.
echo -1000 | sudo tee /proc/1234/oom_score_adj

To make this persistent for a Docker container, you can pass it via docker run or docker-compose:

# For docker run:
sudo docker run -d --name my-web-server --oom-score-adj -500 --memory="4096m" my-app:latest

# For docker-compose.yml:
version: '3.8'
services:
  my-web-server:
    image: my-app:latest
    ports:
      - "80:80"
    oom_score_adj: -500 # Adjust OOM killer priority
    deploy:
      resources:
        limits:
          memory: 4096m

Using oom_score_adj or oom-kill-disable (which completely disables OOM killing for the container by setting oom_score_adj to -1000) for a container can be dangerous. If this container truly exhausts all host memory, it can lead to a kernel panic or an unresponsive system. Use with extreme caution and only for services where you've extensively profiled memory usage and are confident it won't consume the entire system's resources.

7. Monitor and Alert

Implement robust monitoring for your Docker containers and host system.

• Docker Stats Exporter: Use Prometheus cadvisor or node_exporter with Docker collectors to export container and host metrics (CPU, memory, disk I/O) to Prometheus.
• Grafana Dashboards: Visualize these metrics in Grafana to easily identify trends, memory spikes, and predict potential OOM events.
• Alerting: Set up alerts in Prometheus Alertmanager (or your chosen monitoring system) for high memory utilization, container restarts, or OOM-specific messages in dmesg/journalctl.

Proactive monitoring allows you to identify resource bottlenecks before they lead to service disruption.