Troubleshooting Docker Container Exited with Code 137 (OOM Killed) on Ubuntu 22.04 LTS

Resolve Docker containers exiting with code 137 (Out Of Memory) on Ubuntu 22.04 LTS. Learn to diagnose and fix OOM kills effectively.

When a Docker container unexpectedly stops with an exit code of 137, it's a strong indicator that the operating system's Out-Of-Memory (OOM) killer has terminated the container due to resource exhaustion. This typically means the container, or the host system itself, ran out of available memory, leading to instability or service interruptions for the application running within that container. Understanding the root cause is crucial for a stable, high-performance Docker environment.

Symptom & Error Signature

Users will typically observe their Docker containers stopping abruptly. Attempts to restart them might succeed temporarily before they crash again, or they might fail to start if the memory pressure is immediate.

You might see the following when checking container status:

sudo docker ps -a

CONTAINER ID   IMAGE                 COMMAND                  CREATED         STATUS                      PORTS     NAMES
a1b2c3d4e5f6   my-app:latest         "npm start"              5 minutes ago   Exited (137) 2 minutes ago            my-web-app

Checking the container logs might not always reveal the OOM event, as the kernel kills the process before it can log. However, you might see the application terminating without a graceful shutdown:

sudo docker logs a1b2c3d4e5f6

...
[INFO] Application started on port 8080
[ERROR] Database connection failed after 5 retries.
(application logs might abruptly stop here without explicit error messages related to memory)
...

The most definitive evidence of an OOM kill comes from the host system's kernel logs:

dmesg -T | grep -i 'oom|killed'

[Tue Oct 24 10:30:05 2026] node:0 zone:DMA32 flags:0x0
[Tue Oct 24 10:30:05 2026] oom-kill:constraint=CONSTRAINT_NONE, node=0,
[Tue Oct 24 10:30:05 2026] Out of memory: Killed process 1234 (node) total-vm:4194304kB, anon-rss:2097152kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:2048kB oom_score_adj:0
[Tue Oct 24 10:30:05 2026] oom_reaper: reaped process 1234 (node), now dead
[Tue Oct 24 10:30:05 2026] Memory cgroup out of memory: Killed process 5678 (java) total-vm:8388608kB, anon-rss:4194304kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:4096kB oom_score_adj:0

Also, journalctl can provide insights into the Docker daemon's perspective, though it usually just reports the exit:

sudo journalctl -u docker.service -r | head -n 20

Oct 24 10:30:05 your_hostname dockerd[987]: time="2026-10-24T10:30:05.123456789Z" level=info msg="Container a1b2c3d4e5f6 exited with code 137"
Oct 24 10:30:05 your_hostname dockerd[987]: time="2026-10-24T10:30:05.123456789Z" level=info msg="ignoring event" container="a1b2c3d4e5f6" module=libcontainerd namespace=moby topic="/containers/delete"

Root Cause Analysis

The core reason for an exit code 137 is the container process being terminated by the Linux Out-Of-Memory (OOM) killer. This occurs when the system, or a specific cgroup (which Docker uses for resource limiting), runs out of memory. Several factors can contribute to this:

1. Insufficient Host System Memory: The physical RAM on the server is simply not enough to accommodate all running processes, including the Docker daemon and its containers, other system services (e.g., database, web server), and the host operating system itself.
2. Container Memory Leak or High Usage: The application running inside the Docker container is consuming more memory than expected. This could be due to:
   • Memory leaks in the application code.
   • Inefficient data structures or algorithms.
   • Uncapped caches or buffer sizes.
   • Spikes in workload leading to temporary but significant memory demands.
   • Misconfigured application settings (e.g., JVM heap size, PHP memory_limit, Node.js old space size) causing it to attempt to use more memory than available or allowed.
3. Restrictive Docker Memory Limits: Docker allows you to set memory limits for individual containers using the --memory (-m) or mem_limit (in Docker Compose) options. If these limits are set too low for the application's actual needs, the container's cgroup will trigger an OOM kill even if the host has plenty of free memory.
4. Lack of Swap Space or Aggressive Swappiness: While often discouraged for performance-critical applications, swap space can provide a safety net for burst memory usage. If swap is disabled or insufficient, the kernel has fewer options before resorting to the OOM killer. An overly aggressive vm.swappiness setting (defaults to 60 on Ubuntu) might also cause excessive swapping, degrading performance and potentially leading to OOM if swap itself is exhausted.

Step-by-Step Resolution

Addressing OOM issues requires a systematic approach, starting with diagnosis and moving towards targeted solutions.

1. Confirm the OOM Kill Event

First, ensure the container was indeed OOM killed.

# Check dmesg for OOM killer messages
sudo dmesg -T | grep -i 'oom|killed' | tail -n 20

# Inspect the container's exit code and details
sudo docker inspect <container_id_or_name> | grep -E "ExitCode|OOMKilled"

Look for ExitCode: 137 and OOMKilled: true in the inspect output. If OOMKilled is false but the exit code is 137, it could indicate that the process itself received a SIGKILL (signal 9) for another reason, but OOM killer is the most common cause.

2. Analyze Container Memory Usage Patterns

Understanding how much memory your application needs is critical.

• Real-time Monitoring:

  sudo docker stats <container_id_or_name>
  

  Observe the MEM USAGE / LIMIT column. If MEM USAGE consistently approaches LIMIT (or the host's total memory if no limit is set), you're nearing OOM territory.

• Historical Monitoring: Implement monitoring solutions like Prometheus with cAdvisor, Grafana, or a commercial APM tool (Datadog, New Relic) to collect and visualize historical memory usage for your containers and host. This helps identify trends, peaks, and potential memory leaks over time.

• Inside the Container (if accessible):

  # Open a shell into the running container (if it stays up long enough)
  sudo docker exec -it <container_id_or_name> sh
  # Or, if using Bash
  sudo docker exec -it <container_id_or_name> bash
  
  # Inside the container:
  free -h
  top -o %MEM
  

  This gives you a view of memory from within the container's perspective. Keep in mind that tools like free inside a cgroup-limited container might show total host memory if not properly container-aware, so focus on RES (Resident Set Size) for individual processes.

• Application Profiling: For complex applications, use language-specific profiling tools (e.g., Java Flight Recorder, Python memory_profiler, Node.js heap snapshots) to pinpoint memory-intensive code sections or leaks.

3. Adjust Docker Memory Limits

If your application consistently needs more memory than its current Docker limit, increase it.

• For docker run commands: Use the --memory (-m) and --memory-swap flags. mem_limit defines the hard memory limit. memswap_limit defines the total memory (RAM + swap) a container can use. Setting --memory-swap to -1 (the default) means the container can use as much swap as the host system has configured, in addition to its RAM limit.

  # Example: Allow up to 2GB RAM, unlimited host swap
  sudo docker run -it --name my_app_container -m 2g --memory-swap -1 my_image:latest
  

• For docker-compose.yml: Define mem_limit and memswap_limit under the service definition.

  version: '3.8'
  services:
    my_app:
      image: my_image:latest
      container_name: my_app_container
      ports:
        - "80:8080"
      deploy:
        resources:
          limits:
            memory: 2g
          reservations: # Optional: Soft limit, container might temporarily exceed up to hard limit
            memory: 1g
      # mem_swap_limit is a top-level property under service, not deploy.resources
      # Use memswap_limit for total memory (RAM + swap)
      mem_limit: 2g          # Hard limit for RAM
      memswap_limit: 3g      # Total memory (RAM + Swap) allowed. If -1, unlimited swap from host.
                             # If memswap_limit is omitted, it defaults to mem_limit * 2.
                             # Setting it to 0 means no swap usage.
  

  Increasing memory limits without first understanding why the application is consuming so much memory can merely postpone the problem or shift the OOM to another container or the host itself. Always pair limit adjustments with analysis.

4. Optimize Application Memory Usage

The most robust solution is to make the application more memory-efficient.

• Code Review: Identify and fix memory leaks or inefficient resource management.
• Configuration Tuning:
  • JVM Applications: Adjust -Xmx (max heap size) and -Xms (initial heap size). Ensure -Xmx is less than your Docker mem_limit. Consider JVM garbage collector tuning.
  • Node.js: Monitor heap usage, identify memory-intensive closures.
  • PHP: Adjust memory_limit in php.ini or .htaccess.
  • Databases (e.g., MySQL, PostgreSQL): Tune buffer pool sizes, cache sizes (innodb_buffer_pool_size for MySQL). Ensure these fit within the container's memory limit.
• Resource Management: Ensure proper closing of file handles, database connections, and network sockets.

5. Increase Host System Memory

If multiple containers are consistently hitting their limits or the host itself is showing high memory usage even with optimized containers, the server might simply be under-provisioned.

• Upgrade the server's RAM. This is a direct but potentially costly solution.
• Distribute workloads across multiple smaller hosts.

Always assess if the host memory is truly the bottleneck before investing in hardware upgrades. Sometimes, a single runaway process or inefficient application configuration is the culprit.

6. Configure Swap Space (If Applicable and Necessary)

While not a substitute for sufficient RAM, configuring swap space can help the kernel manage memory pressure during temporary spikes, potentially preventing an OOM kill.

# Check current swap status
sudo swapon --show

# Create a 4GB swap file (adjust size as needed)
sudo fallocate -l 4G /swapfile

# Set appropriate permissions
sudo chmod 600 /swapfile

# Set up the swap area
sudo mkswap /swapfile

# Enable the swap file
sudo swapon /swapfile

# Make swap persistent across reboots by adding to /etc/fstab
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

# Adjust swappiness (lower value means less aggressive swapping, e.g., 10-20 for servers)
# Check current swappiness
# cat /proc/sys/vm/swappiness
sudo sysctl vm.swappiness=10
# Make swappiness persistent
echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p # Apply changes immediately

Excessive swapping severely degrades system and application performance. Swap should be used as a last resort or for handling occasional memory bursts, not as a primary solution for chronic memory shortages.

7. Monitor and Alert

Implement comprehensive monitoring for both host and container memory usage.

• Host-level: Monitor total RAM usage, swap usage, and CPU load.
• Container-level: Monitor individual container memory, CPU, and network I/O.
• Set up alerts for:
  • High memory utilization (e.g., >80% RAM usage on host or container).
  • OOM killer activations (by parsing dmesg or syslog).
  • Container restarts or unhealthy states.

Tools like Prometheus, Grafana, cAdvisor, Node Exporter, or commercial monitoring solutions are invaluable here.

8. Review and Adjust OOM Killer Tuning (Advanced)

This is an advanced step and should be approached with caution. Generally, default OOM killer behavior is sensible.

• vm.oom_kill_allocating_task: (Default: 0) If set to 1, the OOM killer will prioritize killing the task that triggered the OOM event. Setting it to 0 allows the OOM killer to select the "best" process to kill, based on oom_score.
• vm.oom_dump_tasks: (Default: 1) If set to 1, the OOM killer will dump a list of current tasks and their OOM scores, which is useful for debugging.
• oom_score_adj: You can adjust the oom_score_adj for specific processes to make them more or less likely to be killed. This is usually done for critical host services, not typically for Docker containers themselves as Docker manages its cgroups and OOM scores.

Modifying kernel OOM killer parameters can have significant and potentially negative impacts on system stability. Only proceed with a deep understanding of the implications and thorough testing.

By following these steps, you can effectively diagnose, troubleshoot, and resolve Docker container OOM kill issues on your Ubuntu 22.04 LTS servers, leading to more stable and reliable deployments.