Troubleshooting Docker Container Exit Code 139 (Segmentation Fault) on Debian 12 Bookworm

Resolve Docker containers exiting with code 139 (segmentation fault) on Debian 12 Bookworm by diagnosing memory, glibc, and application-level issues.

When a Docker container unexpectedly terminates with an exit code of 139, it signifies a SIGSEGV signal, commonly known as a "Segmentation Fault." This critical error indicates that the program attempted to access a memory location that it was not allowed to access, or tried to access a memory location in a way that is not allowed (e.g., writing to a read-only location). On Debian 12 Bookworm, this issue can be particularly nuanced due to its specific glibc version and kernel characteristics. This guide provides an expert-level approach to diagnosing and resolving such an issue.

Symptom & Error Signature

Users typically observe their Docker container failing to start or crashing shortly after launch. Attempts to restart the container will also fail. The most direct indication of a segmentation fault comes from docker ps -a and docker logs.

Example docker ps -a output:

CONTAINER ID   IMAGE                 COMMAND                  CREATED         STATUS                          PORTS     NAMES
a1b2c3d4e5f6   my-problem-app:latest "/usr/bin/python3 app.py" 2 minutes ago   Exited (139) 1 minute ago             my-web-app

Example docker logs <container_name> output:

... (previous application logs) ...
Segmentation fault

Additional Host System Logs (via dmesg or journalctl):

The Linux kernel might also log information about the segmentation fault, often indicating the process name and potentially the instruction pointer (IP) at the time of the crash.

sudo dmesg | grep -i "segmentation fault"

Output might look like:

[ 1234.567890] my-app[12345]: segfault at 7f8b2c000000 ip 00007f8b2c000000 sp 00007ffc9d0a1234 error 4 in libsomelib.so[7f8b2c000000+1000]

Or via journalctl:

sudo journalctl -xe | grep -i "segmentation fault"

Root Cause Analysis

A segmentation fault (exit code 139) points to a memory access violation. The root causes in a Dockerized environment on Debian 12 can vary but commonly fall into these categories:

1. Application-Level Bugs: This is the most common cause.

   • Memory Corruption: C/C++ applications or those using native extensions (e.g., Python with ctypes, Node.js with native addons) often encounter segfaults due to null pointer dereferences, buffer overflows/underflows, use-after-free errors, or attempts to write to read-only memory.
   • Race Conditions: Concurrency issues in multi-threaded applications can lead to corrupted shared memory and subsequent segfaults.
   • Incorrect Library Usage: Misuse of external libraries or FFI (Foreign Function Interface) can lead to invalid memory operations.

2. Resource Exhaustion (OOM – Out Of Memory): While the Linux OOM Killer usually terminates processes with a SIGKILL (exit code 137), severe memory pressure or incorrect memory allocation strategies within an application can sometimes lead to a segfault before the OOM Killer intervenes, especially if the application tries to access memory that has been reclaimed or was never properly allocated.

   • Insufficient Container Memory Limits: The Docker container is simply not allocated enough RAM to run the application correctly.
   • Memory Leaks: The application continuously allocates memory without freeing it, eventually exhausting available resources.

3. glibc Version Incompatibility: Debian 12 "Bookworm" ships with glibc 2.36. This is a critical point.

   • Host-Container Mismatch: If your Docker container image was built on a significantly older or different distribution (e.g., Ubuntu 20.04 with glibc 2.31, or a very specific centos/alpine variant with musl libc) and dynamically links to libraries that behave differently with glibc 2.36, it can lead to memory layout issues, symbol resolution problems, or unexpected library function behavior, resulting in a segfault. This is especially prevalent if pre-compiled binaries are used without careful environment matching.
   • Static Linking Issues: While less common, some statically linked binaries might still rely on specific kernel syscall interfaces or features that could interact unexpectedly with the host kernel or Docker runtime environment.

4. Corrupt Libraries or Binaries: Though rare, a corrupted application binary or a shared library (.so file) inside the container could cause an instruction pointer to jump to an invalid memory address. This can stem from filesystem corruption during image build/transfer or storage issues on the host.

5. Hardware Issues: Extremely rare, but faulty RAM on the host machine can cause memory corruption detectable as a segmentation fault by any running process, including Docker containers.

Step-by-Step Resolution

Debugging segmentation faults requires a systematic approach, often involving inspecting logs, resource usage, and potentially deep-diving into the application's runtime.

1. Analyze Docker & Host System Logs

Start by gathering all available log data.

1. Check Container Logs:

   docker logs <container_name_or_id>
   

   Look for specific messages leading up to the Segmentation fault line. Sometimes, the application itself might log an internal error or stack trace before crashing.

2. Inspect Host Kernel Logs:

   sudo dmesg -T | grep -i "segfault"
   

   This shows kernel-level reports of segmentation faults, including the process name, PID, and possibly the faulting address. This is crucial for confirming the crash origin.

3. Review Systemd Journal for Docker Service:

   sudo journalctl -u docker.service -r --since "5 minutes ago"
   

   Look for any Docker-specific errors or warnings around the time the container crashed.

2. Verify and Adjust Container Resource Limits

Insufficient memory is a frequent culprit.

1. Inspect Current Container Limits:

   docker inspect <container_name_or_id> | grep -E "Memory|Cpu"
   

   Pay close attention to Memory and MemorySwap settings under HostConfig.

2. Monitor Live Resource Usage:

   docker stats <container_name_or_id>
   

   Run this command before the container crashes (if possible) or during a period of high load to see if it's hitting memory limits.

3. Increase Memory and Swap Limits: If resource exhaustion is suspected, try allocating more memory.

   Increasing memory limits can consume more host resources. Ensure your host has sufficient free RAM.

   If restarting the container:

   docker stop <container_name>
   docker rm <container_name>
   docker run -d --name <container_name> --memory "1g" --memory-swap "2g" <image_name>
   

   If updating a running container (Docker Engine 1.13+):

   docker update --memory "1g" --memory-swap "2g" <container_name>
   

   Start with a moderate increase (e.g., from 512MB to 1GB), then test.

3. Analyze glibc Version Compatibility

glibc mismatches are a common but often overlooked cause of segfaults, especially on newer Debian versions like Bookworm.

1. Determine Host glibc Version (Debian 12):

   ldd --version
   # Expected output: ldd (Debian GLIBC 2.36-9+deb12u1) 2.36
   

   Or check package version:

   apt-cache policy libc6
   # Expected output: Installed: 2.36-9+deb12u1
   

2. Determine Container glibc Version: This requires entering the container, if it can run long enough, or running a temporary debug container.

   docker exec -it <container_name_or_id> ldd --version
   

   If ldd is not found, you might need to install libc-bin or build-essential inside the container temporarily (e.g., for Debian/Ubuntu-based images: apt update && apt install -y libc-bin).

   Installing development packages like build-essential in a production image is generally discouraged. Use a multi-stage build or a temporary debug container.

3. Resolve glibc Mismatches:

   • Preferred Solution: Rebuild with Matching Base Image: The most robust solution is to use a container base image that closely matches your host's glibc version. For Debian 12, this means using debian:bookworm-slim or debian:bookworm. If using Ubuntu, ubuntu:22.04 might also be compatible due to similar glibc versions.
   • Alternative: Use Fully Static Binaries (if applicable): If your application can be compiled with all dependencies statically linked, it can mitigate glibc version issues. However, this isn't always feasible and can result in larger binaries.
   • Consider Alpine for Minimal Images: If glibc is consistently problematic, consider switching to an Alpine Linux base image if your application can be compiled against musl libc. This significantly reduces image size and dependency surface but requires recompilation.

4. Debug Application Inside the Container

If the problem persists, it's likely an application-level bug.

1. Access the Container Shell:

   docker exec -it <container_name_or_id> /bin/bash
   

   If the container crashes immediately, you might need to modify your Dockerfile CMD or ENTRYPOINT to start a shell instead of your application:

   # Temporarily
   CMD ["/bin/bash"]
   

   Then docker run -it my-problem-app:latest /bin/bash.

2. Install Debugging Tools (Temporarily): Inside the container, install tools like gdb, strace, or valgrind.

   apt update && apt install -y gdb strace valgrind
   

   Do NOT deploy production images with debugging tools installed, as they increase image size and potential attack surface. Use a multi-stage build approach for debugging stages.

3. Run Application with strace: strace logs system calls and signals, which can show what the application was doing just before the crash.

   strace -f -o /tmp/app.log <your_application_command>
   

   Review /tmp/app.log for SIGSEGV or unusual syscall patterns.

4. Run Application with gdb (GNU Debugger): If your application is compiled with debugging symbols (-g flag for C/C++), gdb can provide a precise backtrace.

   gdb --batch -ex "run" -ex "bt full" -ex "info registers" -ex "q" --args <your_application_command_and_args>
   

   The output will include a detailed stack trace, showing the function calls leading up to the segfault. This is invaluable for pinpointing the exact line of code.

5. Run Application with valgrind: valgrind (specifically memcheck) is excellent for detecting memory errors like invalid reads/writes, use-after-free, and memory leaks. It adds significant overhead.

   valgrind --tool=memcheck --leak-check=full --show-leak-kinds=all --track-origins=yes <your_application_command>
   

   Analyze valgrind's output for reported errors.

5. Consider Filesystem Corruption

While less likely for a segfault, ensure the container's filesystem layers aren't corrupted.

1. Pull Image Again:

   docker pull <image_name>
   

   This ensures you have a fresh, uncorrupted copy of the image.

2. Rebuild Image: If you build your own images, force a rebuild to ensure all layers are fresh.

   docker build --no-cache -t my-app:latest .
   

6. Examine Host Hardware (Last Resort)

If all software-related debugging fails and other applications on the same host are also exhibiting strange behavior, consider host hardware issues.

1. Check dmesg for MCE Errors:

   sudo dmesg | grep -i "mce"
   

   Machine Check Exception (MCE) errors often indicate CPU or memory hardware problems.

2. Run Memory Diagnostics: On a physical server, schedule a memtest86+ run. For virtual machines, ensure the underlying hypervisor host is healthy and its memory is stable.

By systematically working through these steps, you should be able to diagnose and resolve the "Docker container exited with code 139 segmentation fault" error on your Debian 12 Bookworm system.