Troubleshooting Docker Container Exit Code 139 (Segmentation Fault) on CentOS Stream / Rocky Linux

Resolve Docker containers exiting with code 139 (segmentation fault) on CentOS Stream or Rocky Linux. This guide covers common causes and provides a step-by-step resolution.

When a Docker container abruptly stops with an "Exited (139)" status, it indicates a Segmentation Fault (SIGSEGV). This critical error signifies that a process within the container attempted to access a memory location it was not authorized to access, or tried to access it in an invalid way. On CentOS Stream or Rocky Linux hosts, this often points to issues with memory allocation, application bugs, or system resource limitations impacting the container's runtime environment.

Symptom & Error Signature

The most immediate symptom is a Docker container failing to start or crashing unexpectedly shortly after launch. Your application service will be unavailable. You'll typically observe the Exited (139) status when listing containers and may see "Segmentation fault (core dumped)" in the container or host logs.

Checking container status:

[root@host ~]# docker ps -a
CONTAINER ID   IMAGE           COMMAND                  CREATED          STATUS                       PORTS     NAMES
abc123def456   my-app:latest   "/usr/local/bin/entry"   5 seconds ago    Exited (139) 4 seconds ago             my-app-container

Inspecting container logs:

[root@host ~]# docker logs abc123def456
# ... (Potential application output before crash) ...
Segmentation fault (core dumped)

Checking host system logs for segfaults or OOM killer events:

[root@host ~]# journalctl -xe | grep -i "segfault|fault|coredump|oom-killer"

Or specific kernel messages:

[root@host ~]# dmesg -T | grep -i "segfault|fault|coredump"
[Mon Jan 01 12:34:56 2024] my-app[12345]: segfault at 7f1234567890 ip 00007f1234567890 sp 00007ffc98765432 error 4 in libmylib.so[7f1234567000+1000]
[Mon Jan 01 12:34:56 2024] Code: 48 83 ec 08 e8 a0 1b ff ff c9 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 48 8d 3c 25 <48> 00 00 00 00 48 8b 07 48 8b 7f 08 48 8d 3c 25 00 00 00 00 48 8b

Root Cause Analysis

A Segmentation Fault (SIGSEGV, signal 11) indicates a memory access violation. In the context of Docker containers on a CentOS Stream / Rocky Linux host, the root causes can be multifaceted:

1. Memory Exhaustion (OOM – Out Of Memory) within cgroups: This is arguably the most common cause for SIGSEGV in containers. When the container's memory usage hits its hard limit (defined by Docker's --memory flag or host cgroups), the kernel's OOM killer might intervene. Rather than always sending a SIGKILL (signal 9), processes can sometimes encounter a SIGSEGV if they attempt to allocate or access memory that has been revoked or made unavailable by the kernel in a pre-OOM state, or if the kernel's memory management becomes inconsistent under extreme pressure.

2. Application Bug: The application running inside the container has a bug related to memory management. This is prevalent in compiled languages like C, C++, Rust, or Go, where direct memory access is common. Examples include:

   • Dereferencing a null pointer.
   • Buffer overflow/underflow (writing outside allocated memory bounds).
   • Use-after-free errors.
   • Accessing uninitialized memory.
   • Stack overflow due to deep recursion or large stack allocations. While Python, Java, Node.js applications usually throw exceptions, native extensions (e.g., C/C++ libraries called by Python via ctypes or pybind11) can still trigger segfaults.

3. Host Kernel / Glibc Incompatibility: A significant mismatch between the glibc version used to compile the application within the container and the host system's kernel or libraries can lead to unexpected behavior, including segfaults, especially with system calls or highly optimized libraries. CentOS Stream and Rocky Linux use newer kernels and glibc versions, which usually offer good compatibility, but issues can arise with very old or custom-built container images.

4. Corrupt Docker Image or Storage Issue: A corrupted Docker image layer, an underlying filesystem issue on the host, or a problem with the Docker storage driver can lead to binaries or libraries being loaded incorrectly, resulting in memory access violations.

5. Insufficient Stack Size: The default stack size (ulimit -s) within the container might be too small for the application's needs, leading to a stack overflow and subsequent segfault.

6. Hardware Fault: While less common, faulty RAM or CPU on the host machine can manifest as seemingly random segfaults in processes, including those running inside containers.

Step-by-Step Resolution

Debugging a SIGSEGV requires a systematic approach, often starting with resource checks and then moving into application-specific diagnostics.

1. Analyze Docker Container Logs and Host System Logs

The first step is always to gather as much information as possible from the logs.

1. Check Docker logs for the specific container:

   docker logs <container_id_or_name>
   

   Look for any output immediately preceding "Segmentation fault (core dumped)". It might provide hints about which part of the application was running.

2. Inspect host system logs for kernel messages:

   journalctl -u docker -f # Follow Docker daemon logs
   journalctl -xe | grep -i "segfault|fault|coredump|oom-killer" # Search for relevant kernel messages
   dmesg -T | grep -i "segfault" # Check kernel ring buffer for segfault details
   

   Pay close attention to messages from the kernel's OOM killer. If you see Memory cgroup out of memory, it strongly suggests a memory limit issue. The dmesg output can sometimes pinpoint the exact library or function causing the fault.

2. Review and Adjust Docker Memory & Resource Limits

Insufficient memory is a primary suspect.

1. Check current container resource limits:

   docker inspect <container_id_or_name> | grep -E "Memory|CpuShares|KernelMemory"
   

   Specifically look at Memory and MemorySwap under HostConfig.

2. Test with increased memory limits: If you suspect an OOM issue, try running the container with more memory.

   > [!IMPORTANT]
   > Start with a conservative increase, then progressively raise if the issue persists. Do not allocate an unreasonable amount of memory, as this can starve other processes on your host.
   
   docker stop <container_id_or_name>
   docker rm <container_id_or_name>
   docker run -d --name my-app-container --memory="2G" --memory-swap="4G" my-app:latest
   

   Adjust 2G and 4G (for memory and memory + swap) as needed. Monitor the container and host memory usage (docker stats, free -h) after this change.

3. Examine cgroup settings on the host (advanced): For a deeper dive into cgroup memory usage, you can inspect the cgroup filesystem.

   cat /sys/fs/cgroup/memory/docker/<container_long_id>/memory.usage_in_bytes
   cat /sys/fs/cgroup/memory/docker/<container_long_id>/memory.limit_in_bytes
   cat /sys/fs/cgroup/memory/docker/<container_long_id>/memory.failcnt
   

   A rapidly increasing memory.failcnt confirms memory pressure.

3. Analyze Application Code for Memory Errors

If resource limits are not the issue, the problem likely lies within the application itself.

1. Run with debug tools (if applicable): If your application uses C/C++, Rust, or other compiled languages, consider running it with memory debugging tools.

   • Valgrind (for C/C++): This tool detects memory errors like use-after-free, buffer overflows, etc.

     # First, ensure valgrind is installed in your Docker image or use a base image with it.
     # Example Dockerfile snippet:
     # RUN yum update -y && yum install -y valgrind && yum clean all
     # CMD ["valgrind", "--leak-check=full", "--show-leak-kinds=all", "--track-origins=yes", "your-app-executable"]
     
     # Or run it interactively:
     docker run -it --rm --name debug-app my-app:latest valgrind --leak-check=full your-app-executable
     

     Valgrind output is verbose but invaluable for finding memory corruption.

   • GDB (GNU Debugger): Attach GDB to a crashing process or analyze a core dump.

     # Ensure gdb and debug symbols are available in your image.
     # Set up core dump collection on the host (e.g., ulimit -c unlimited, enable systemd-coredump)
     # docker run --ulimit core=-1 your-app:latest # allow core dumps inside container
     # Then, on the host, analyze the core dump with gdb
     gdb your-app-executable core_dump_file
     # (gdb) bt full
     # (gdb) frame N
     

2. Isolate the issue:

   • Try running the application outside Docker on a similar CentOS Stream/Rocky Linux host. Does it segfault there?
   • If it's a Python/Node.js/Java application, check for native extensions or libraries that could be causing the issue. Update them or remove them to test if the segfault disappears.

4. Verify Docker Image Integrity and Compatibility

A corrupted image or incompatible base image can also lead to segfaults.

1. Rebuild the Docker image: Clear the Docker build cache and rebuild your image. This ensures all layers are pulled and built fresh.

   docker build --no-cache -t my-app:latest .
   

2. Pull a fresh base image: If your image relies on a base image (e.g., rockylinux/rockylinux:8), pull it again to ensure no local corruption.

   docker pull rockylinux/rockylinux:8
   

3. Check base image and application library compatibility: Ensure the libraries compiled into your application (especially glibc) are compatible with the version provided by your base image. You can use ldd inside a running container for binaries.

   docker exec -it <container_id> ldd /path/to/your/binary
   

   This shows dynamic library dependencies. Incompatible glibc versions can cause subtle issues.

5. Adjust Ulimits (User Limits) within the Container

An insufficient stack size or too few open file descriptors can sometimes lead to a segfault.

1. Check current ulimits inside a running container (if possible before crash):

   docker exec -it <container_id> ulimit -a
   

   Pay attention to stack size (bytes, -s) and open files (-n).

2. Increase relevant ulimits: If the default stack size (8192 Kbytes) or number of open files is too low for your application's workload, you can specify higher limits during docker run.

   docker stop <container_id_or_name>
   docker rm <container_id_or_name>
   docker run -d --name my-app-container 
     --ulimit stack=16384:16384 
     --ulimit nofile=65536:65536 
     my-app:latest
   

   stack=16384:16384 sets a hard and soft limit of 16MB for the stack. nofile=65536:65536 sets 65536 as the limit for open files. Adjust these values based on your application's requirements.

6. Examine Host System Resources & Health

While rare, host hardware or filesystem issues can indirectly cause container instability.

1. Check host memory and disk:

   free -h
   df -h
   

   Ensure your host isn't critically low on RAM or disk space, as this can impact Docker's ability to manage containers and their data.

2. Test host RAM: If segfaults appear random and affect multiple containers or even host processes, consider running a memory diagnostic tool like memtest86+ (requires rebooting the host) to rule out faulty RAM.

7. Recreate Docker Environment

As a last resort, if all else fails, a clean slate can sometimes resolve obscure issues.

1. Stop and remove all containers:

   docker stop $(docker ps -aq)
   docker rm $(docker ps -aq)
   

2. Remove all Docker images, volumes, and networks (use with caution!):

   > [!WARNING]
   > `docker system prune -a` will delete ALL stopped containers, ALL unused networks, ALL dangling images, ALL build cache, and ALL unused volumes. Ensure you have backups of any critical data in volumes before proceeding.
   
   docker system prune -a --volumes
   

3. Restart the Docker daemon:

   systemctl restart docker
   

   Attempt to run your container again after this. This clears any potential corrupted daemon state or temporary files.