Troubleshooting Docker Container Exit Code 137 (OOM Killed) on Debian 12 Bookworm

Resolve Docker container exit code 137 on Debian 12 due to OOM kills. Learn to diagnose memory limits and optimize resource allocation for stable operations.

When a Docker container unexpectedly stops with an exit code of 137, it's a clear signal from the Linux kernel: the container was terminated by the Out-Of-Memory (OOM) killer. This critical issue means your container attempted to allocate more memory than it was allowed or available, leading to its abrupt termination to protect the overall stability of the host system. For users, this translates to an unresponsive or crashing application, potentially causing service outages.

Symptom & Error Signature

The most common symptom is your Docker container repeatedly failing to start or stopping abruptly. When checking the status of your containers, you will observe the Exited (137) status:

sudo docker ps -a

CONTAINER ID   IMAGE           COMMAND                  CREATED          STATUS                          PORTS     NAMES
a1b2c3d4e5f6   my-app:latest   "node server.js"         5 minutes ago    Exited (137) 3 minutes ago                my-web-app

Attempts to inspect the container logs (docker logs <container_name>) may yield little information about the crash itself, as the application process didn't have a chance to log its termination before being killed by the kernel. The definitive evidence of an OOM kill typically resides in the host system's kernel logs:

sudo dmesg -T | grep -i "oom-killer" | tail -n 20

[Thu Jan 1 12:34:56 2026] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0
[Thu Jan 1 12:34:56 2026] oom-kill:gfp_mask=0x140cgs-0x200000(GFP_KERNEL|__GFP_COMP), order=0, oom_score_adj=0
[Thu Jan 1 12:34:56 2026] docker-a1b2c3d4e:gfp_mask=0x140cgs-0x200000(GFP_KERNEL|__GFP_COMP), order=0, oom_score_adj=0
[Thu Jan 1 12:34:56 2026] Memory cgroup out of memory: Killed process 12345 (node) total-vm:4567890kB, anon-rss:987654kB, file-rss:12345kB, shmem-rss:0kB
[Thu Jan 1 12:34:56 2026] oom_reaper: reaped process 12345 (node), now dead

You might also find relevant entries in journalctl:

sudo journalctl -xe | grep -i "oom"

Root Cause Analysis

Exit code 137 signifies that the container process received signal 9 (SIGKILL), combined with a standard exit status offset of 128 (128 + 9 = 137). SIGKILL is an uncatchable signal, meaning the process cannot handle it gracefully and is immediately terminated by the kernel. This is precisely what the Linux OOM (Out-Of-Memory) killer does.

The OOM killer is a kernel mechanism designed to prevent system instability when memory resources are exhausted. When the system (or a specific cgroup, which Docker uses for resource limiting) runs out of available memory, the OOM killer selects a process to terminate, aiming to free up memory and keep the system responsive. In the context of Docker, this typically occurs due to one of the following reasons:

1. Container-Specific Memory Limit Exceeded: Your Docker container has an explicit memory limit (e.g., set via --memory or -m flags in docker run or deploy.resources.limits.memory in Docker Compose). The application inside the container attempted to consume more RAM than this configured limit.
2. Host System Memory Exhaustion: Even without explicit container memory limits, if the sum of all running processes (including all containers and host processes) exceeds the available physical RAM on the host system, the OOM killer may target a memory-hungry container. This is particularly true if the host has insufficient or no swap space configured.
3. Memory Leak or High Demand: The application running inside the container might have a memory leak, causing its memory footprint to grow continuously until it hits a limit. Alternatively, a sudden spike in demand or a computationally intensive operation could lead to a temporary but significant increase in memory usage.
4. Cgroups Version Differences (Debian 12): Debian 12 (Bookworm) uses cgroups v2 by default. Docker Engine supports cgroups v2, but subtle differences in how memory accounting and limits are enforced compared to cgroups v1 might sometimes lead to unexpected OOM behavior if configurations are not precisely tuned or if older images/runtimes are used that implicitly expect v1 behavior.

Step-by-Step Resolution

Addressing an OOM kill requires a methodical approach, starting with diagnosis and moving towards resource adjustment or application optimization.

1. Verify OOM Killer Invocation and Target

Confirm that the OOM killer was indeed responsible and identify which process was targeted within the container's cgroup.

# Check recent kernel logs for OOM events
sudo dmesg -T | grep -i "oom-killer" | tail -n 20

# Check Docker service logs for OOM related messages (less common for direct kills)
sudo journalctl -u docker.service --since "1 hour ago" | grep -i "oom"

Look for lines similar to Memory cgroup out of memory: Killed process <PID> (<process_name>) referencing your container's processes. This confirms the OOM kill and the specific process that was terminated.

2. Inspect Container Resource Configuration

Check the Docker configuration for your container to see if explicit memory limits are set.

sudo docker inspect <container_name_or_id> | grep -E "Memory|MemorySwap"

Example output:

        "Memory": 1073741824,   // 1GB
        "MemorySwap": 2147483648, // 2GB (1GB RAM + 1GB Swap)

• Memory: The maximum amount of RAM the container can use.
• MemorySwap: The total amount of memory (RAM + swap) the container can use. If MemorySwap is equal to Memory, it implies no dedicated swap space for the container itself. If MemorySwap is -1, the container can use unlimited host swap.

You can also monitor the container's real-time memory usage with docker stats if it's able to run for a short period before crashing.

sudo docker stats <container_name_or_id>

3. Analyze Application Memory Profile

If your application has a memory leak or an unexpected high memory demand, adjusting Docker limits will only be a temporary fix.

• Application Logs: Check if your application logs any memory usage statistics or warnings before termination.

• Profiling (if possible): If you can temporarily run the container or a similar test environment, use application-specific profiling tools (e.g., Node.js heap snapshots, Java profilers, Python memory_profiler) to identify memory-intensive operations or leaks.

• Execute inside container: If the container starts briefly, try to docker exec into it and use tools like free -h or top to quickly see memory usage.

  sudo docker exec -it <container_name> /bin/bash
  free -h
  

4. Adjust Docker Container Memory Limits

This is often the first and most direct solution if your application genuinely requires more memory than allocated.

For docker run commands:

To set memory and swap limits:

• --memory="<AMOUNT>" or -m "<AMOUNT>": Sets the hard memory limit for the container (e.g., 2g, 512m).
• --memory-swap="<AMOUNT>": Sets the total memory (RAM + swap) limit.
  • If memory-swap is set to the same value as memory, the container will have no dedicated swap space.
  • If memory-swap is larger than memory, the difference is the amount of swap the container can use.
  • If memory-swap is -1 (default if --memory is set without --memory-swap), the container can use unlimited host swap.

When increasing memory limits, ensure your host machine has sufficient physical RAM to accommodate the new limit plus all other running processes. Over-allocating memory can shift the OOM problem to the host system.

# Example: Allocate 2GB RAM and 0GB swap for the container
sudo docker run -d 
  --name my-web-app 
  --memory="2g" 
  --memory-swap="2g" 
  my-app:latest

# Example: Allocate 2GB RAM and allow 1GB swap (total 3GB)
sudo docker run -d 
  --name my-web-app 
  --memory="2g" 
  --memory-swap="3g" 
  my-app:latest

To update an existing container (requires Docker Engine 1.10+):

# Update an existing container to 2GB RAM and 0GB swap
sudo docker update --memory="2g" --memory-swap="2g" my-web-app

For Docker Compose:

Modify your docker-compose.yml file under the deploy.resources.limits section:

version: '3.8'
services:
  my-web-app:
    image: my-app:latest
    container_name: my-web-app
    deploy:
      resources:
        limits:
          memory: 2G
          # For 0GB swap, set memory_swap equal to memory
          memory_swap: 2G 
          # For 1GB swap (total 3GB), set memory_swap to 3G
          # memory_swap: 3G 

After modifying docker-compose.yml, redeploy your services:

sudo docker compose up -d

5. Increase Host System Memory or Swap

If the entire host system is consistently running low on memory, it's not just a single container's problem but a systemic one.

• Add Physical RAM: The most effective long-term solution is to provision more physical RAM for your server.

• Increase Host Swap Space: As a temporary or supplementary measure, you can increase the host's swap file size.

  # Check current swap usage
  free -h
  
  # Create a new swap file (e.g., 4GB)
  sudo fallocate -l 4G /swapfile
  
  # Set appropriate permissions
  sudo chmod 600 /swapfile
  
  # Make it a swap area
  sudo mkswap /swapfile
  
  # Activate the swap file
  sudo swapon /swapfile
  
  # Add it to /etc/fstab for persistence across reboots
  echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
  
  # Adjust swappiness (optional but recommended for performance)
  # A value of 10-20 is often good for servers, reducing disk I/O from excessive swapping.
  sudo sysctl vm.swappiness=10
  echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf
  

  Relying heavily on swap can severely degrade the performance of your applications and the entire host system, as disk I/O is much slower than RAM. While increasing swap can prevent immediate OOM kills, it's not a substitute for sufficient physical RAM for critical production services.

6. Optimize Application Memory Usage

This is the most robust solution if the issue stems from inefficient application code.

• Code Review: Identify and fix memory leaks or inefficient data structures/algorithms.
• Dependency Audit: Check if any third-party libraries or frameworks are known to be memory-hungry or have leaks.
• Configuration Tuning: Many runtimes (JVM, Node.js, Python) allow tuning of garbage collection parameters. Adjust these to be more aggressive in reclaiming memory, if appropriate.
• Resource Throttling: If your application handles concurrent requests, consider implementing rate limiting or connection pooling to manage peak memory demands more effectively.

7. Monitor and Test

After implementing any changes, it's crucial to monitor your system and containers closely.

• Use sudo docker stats <container_name> to observe real-time memory usage.
• Monitor host memory usage with free -h or htop.
• If you have a monitoring stack (e.g., Prometheus/Grafana, ELK stack), observe memory trends over time.
• Perform load testing to simulate peak usage and ensure stability under expected conditions.