Resolving Kubernetes Pod OOMKilled Errors: Resource Limit Exceeded on Ubuntu 22.04 LTS

Diagnose and fix Kubernetes Pod OOMKilled errors caused by resource limit overages on Ubuntu 22.04. Optimize your cluster for stability and performance.

The OOMKilled status in Kubernetes indicates that a container within your Pod was terminated by the Linux kernel's Out-Of-Memory (OOM) killer. This occurs when a process attempts to consume more memory than its allocated cgroup limit, leading to a forced termination to prevent system instability. On Ubuntu 22.04 LTS, with its adoption of cgroup v2, the enforcement mechanisms are robust, making proper resource configuration paramount for stable containerized applications.

This guide will walk you through diagnosing and resolving OOMKilled errors stemming from resource limit overruns in your Kubernetes environment.

Symptom & Error Signature

When a Pod is OOMKilled, you will typically observe the Pod restarting repeatedly, entering a CrashLoopBackOff state, or simply having a high RESTARTS count. The most definitive signs appear in the Pod's status and events.

You can inspect the Pod's status and events using kubectl:

kubectl get pod <pod-name> -n <namespace>

Example output showing multiple restarts:

NAME         READY   STATUS             RESTARTS   AGE
my-app-pod   0/1     OOMKilled          5          10m

Detailed events will explicitly mention OOMKilled:

kubectl describe pod <pod-name> -n <namespace>

Snippet from kubectl describe pod output:

Name:         my-app-pod
Namespace:    default
Priority:     0
Node:         my-k8s-node01/192.168.1.10
Start Time:   Mon, 26 Oct 2026 10:00:00 +0000
Labels:       app=my-app
Annotations:  <none>
Status:       Running
IP:           10.42.0.5
IPs:
  IP:  10.42.0.5
Containers:
  my-app-container:
    Container ID:  containerd://a1b2c3d4e5f6...
    Image:         my-registry/my-app:1.0
    Image ID:      containerd://g7h8i9j0k1l2...
    Port:          8080/TCP
    Host Port:     0/TCP
    Limits:
      memory:  512Mi
    Requests:
      memory:  256Mi
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137 # Indicates process was killed by an external signal (SIGKILL)
      Started At:   Mon, 26 Oct 2026 10:04:30 +0000
      Finished At:  Mon, 26 Oct 2026 10:04:45 +0000
    Ready:          False
    Restart Count:  5
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-abcde (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-abcde:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From     Message
  ----     ------     ----                  ----     -------
  Normal   Scheduled  9m                    default-scheduler  Successfully assigned default/my-app-pod to my-k8s-node01
  Normal   Pulling    8m (x5 over 9m)       kubelet  Pulling image "my-registry/my-app:1.0"
  Normal   Pulled     8m (x5 over 9m)       kubelet  Successfully pulled image "my-registry/my-app:1.0" in 1.234s (containerID=...)
  Normal   Created    8m (x5 over 9m)       kubelet  Created container my-app-container
  Normal   Started    8m (x5 over 9m)       kubelet  Started container my-app-container
  Warning  OOMKilled  7m (x5 over 8m)       kubelet  Container my-app-container was OOMKilled
  Warning  BackOff    5m (x6 over 8m)       kubelet  Back-off restarting failed container my-app-container in pod my-app-pod_default(a1b2c3d4-e5f6-7890-a1b2-c3d4e5f67890)

The Last State: Terminated with Reason: OOMKilled and Exit Code: 137 is the definitive signature. Exit Code 137 (128 + 9) specifically means the process was terminated by signal 9 (SIGKILL), which is what the OOM killer sends.

Root Cause Analysis

The underlying reasons for an OOMKilled event are generally related to the application's memory consumption exceeding its allocated limits, often within the context of Kubernetes' resource management and the Linux kernel's cgroup enforcement.

1. Insufficient Kubernetes Memory Limits: This is the most prevalent cause. The resources.limits.memory configured in your Pod's manifest is simply too low for the application's actual memory footprint under typical or peak load. Kubernetes uses cgroups to enforce these limits, and when a container exceeds its allocated memory, the kernel's OOM killer steps in.
2. Application Memory Leaks: The application itself might have a memory leak, where it continuously consumes more memory over time without releasing it, eventually hitting any configured limit. This will manifest as a gradual increase in memory usage before the OOMKilled event.
3. Incorrect Memory Requests vs. Limits: While limits are the hard ceiling, requests are used for scheduling and guaranteeing resources. If requests.memory is set unrealistically high compared to limits, or vice-versa, it can lead to inefficient scheduling or premature OOMKills. However, an OOMKilled event specifically points to the limits being hit.
4. Java Virtual Machine (JVM) Heap Size Mismatch: For Java applications, the JVM's default heap size or explicitly configured -Xmx (maximum heap size) might attempt to allocate more memory than the container's memory.limit. Modern JVMs are more container-aware, but explicit configuration remains crucial.
5. Sidecar Container Resource Contention: If a Pod has multiple containers (e.g., application + logging agent sidecar), the combined memory usage or unexpected spikes in a sidecar's usage could lead to one of them hitting its individual limit.
6. Sudden Spikes in Workload: An unexpected surge in traffic or processing load can cause a temporary but significant spike in memory consumption, pushing the application beyond its configured limits.
7. Host Node OOM Conditions (Less Common for Specific Pod OOMKilled): While the kernel's OOM killer prioritizes processes exceeding their cgroup limits, a severely memory-constrained host node can, in extreme cases, lead to more aggressive OOM killing. However, OOMKilled events typically mean the cgroup limit was the primary trigger.

Step-by-Step Resolution

Addressing OOMKilled errors requires a systematic approach involving observation, analysis, and precise configuration adjustments.

1. Identify the OOMKilled Pod and Container

First, confirm which Pods and specific containers within those Pods are experiencing the OOMKilled status.

# List all pods with OOMKilled status across all namespaces
kubectl get pods --all-namespaces | grep OOMKilled

# Get detailed information about the problematic pod
kubectl describe pod <pod-name> -n <namespace>

# Retrieve logs from the *previous* instance of the container (after it was OOMKilled)
kubectl logs <pod-name> -n <namespace> -c <container-name> --previous

The kubectl logs --previous command is invaluable as it allows you to examine what the application was doing just before it was terminated. Look for any error messages, large data processing events, or signs of increasing memory consumption.

2. Analyze Application Memory Usage

Understanding the actual memory footprint of your application is crucial.

• Using kubectl top (Requires Metrics Server): If you have the Kubernetes Metrics Server installed (common in most clusters), you can observe live memory usage.

  # Get overall pod memory usage
  kubectl top pod <pod-name> -n <namespace>
  
  # Get memory usage per container within the pod
  kubectl top pod <pod-name> -n <namespace> --containers
  

  kubectl top shows current usage, which might not reflect peak usage that caused the OOMKill.

• Monitoring Tools (Prometheus/Grafana): Leverage your cluster's monitoring stack (e.g., Prometheus and Grafana) to review historical memory usage trends for the affected Pod/container. Look for patterns:

  • Does memory usage slowly climb over time (potential leak)?
  • Are there sudden spikes correlating with specific events or times of day (peak load)?
  • What was the peak memory usage before the OOMKilled event? This gives you a baseline for setting new limits.

• Local Testing/Profiling: If possible, run the application locally or in a staging environment under similar load conditions. Use profiling tools specific to your application's language (e.g., heapdump for Node.js, go tool pprof for Go, Java VisualVM for Java, memory_profiler for Python) to identify memory-intensive operations or potential leaks.

3. Adjust Kubernetes Resource Limits

Once you have a better understanding of your application's memory requirements, the most common fix is to increase the memory.limits in your Pod, Deployment, or StatefulSet manifest.

When increasing memory limits, aim for a value that is safely above the peak observed memory usage, ideally with a buffer (e.g., 10-20%). Be cautious not to set limits excessively high, as this can lead to inefficient scheduling, increased cloud costs, or mask genuine memory leaks.

1. Edit your Deployment/StatefulSet/Pod manifest: Locate the resources section for the problematic container.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: my-app-deployment
     namespace: default
   spec:
     replicas: 3
     selector:
       matchLabels:
         app: my-app
     template:
       metadata:
         labels:
           app: my-app
       spec:
         containers:
         - name: my-app-container
           image: my-registry/my-app:1.0
           ports:
           - containerPort: 8080
           resources:
             requests:
               memory: "256Mi"
               cpu: "250m"
             limits:
               memory: "512Mi" # <-- INCREASE THIS VALUE (e.g., to "1Gi" or "1.5Gi")
               cpu: "1"
   

2. Apply the changes:

   kubectl apply -f your-deployment.yaml
   

   This will trigger a rolling update, replacing the old Pods with new ones configured with the updated memory limits.

   While increasing limits directly resolves the OOMKilled error, it's crucial to continuously monitor the application's memory usage afterward. If the memory usage continues to climb and eventually hits the new, higher limit, it's a strong indicator of a memory leak in the application itself.

4. Investigate Application Memory Leaks

If simply increasing memory limits doesn't solve the issue, or if you observe memory usage steadily climbing towards the new limit, you likely have a memory leak within your application. This requires debugging the application code.

• Code Review: Review recent code changes, especially those related to data structures, caching, or long-running processes.
• Profiling Tools: Use language-specific profiling tools.
  • Java: VisualVM, JConsole, JProfiler, or jmap/jstat on the host.
  • Node.js: Node.js built-in profiler, heapdump module.
  • Python: memory_profiler, objgraph.
  • Go: go tool pprof (for heap profiles).
• Enable Debugging: For applications that support it, enable verbose logging or debugging modes related to memory management.
• Run on a Test Environment: Deploy the application to a dedicated test environment with high visibility into its runtime metrics and stress-test it to reproduce the leak.

5. Optimizing JVM Memory (If Applicable)

For Java applications, fine-tuning JVM options is critical. The JVM's memory management interacts with cgroup limits.

• Set Explicit Heap Size: Ensure the JVM's maximum heap size (-Xmx) is set to a value less than the container's memory.limit. A good practice is to set -Xmx to approximately 70-80% of the container's memory limit to account for non-heap memory (stack, native libraries, garbage collection overhead, etc.).

  For example, if your container's memory limit is 1Gi:

  containers:
  - name: my-java-app
    image: openjdk:17-jre-slim
    env:
      - name: JAVA_OPTS
        value: "-Xmx768m -Xms256m -XX:MaxRAMPercentage=75.0" # -Xmx768m is 75% of 1Gi
    resources:
      requests:
        memory: "512Mi"
        cpu: "500m"
      limits:
        memory: "1Gi" # Container limit
        cpu: "1"
  

  Modern JVMs (Java 8u191+, Java 10+) are more container-aware and can derive heap sizes from cgroup limits when MaxRAMPercentage is used. However, explicitly setting -Xmx or ensuring MaxRAMPercentage is correctly configured is still the most reliable approach.

6. Verify Node Resource Availability

While OOMKilled primarily points to a container hitting its configured limit, overall node resource scarcity can exacerbate the problem or lead to more aggressive OOM killing.

• Check Node Memory Usage:

  # Check overall node resource usage
  kubectl top nodes
  
  # Get detailed information about the node the pod is running on
  kubectl describe node <node-name>
  

  You can also SSH into the specific node where the OOMKilled Pod was running and check its system-wide memory:

  ssh <node-ip>
  free -h
  htop # Look for overall memory usage and swap activity
  

• Scale Cluster: If nodes are consistently running low on memory, consider adding more nodes to your Kubernetes cluster or upgrading existing nodes with more RAM.

7. Implement Horizontal Pod Autoscaler (HPA) for Memory (Advanced)

If your application's memory usage fluctuates significantly with traffic or workload, configuring an HPA can proactively scale your Pods before they hit memory limits and get OOMKilled. This requires the Kubernetes Metrics Server to be deployed.

The HPA scales based on the average utilization of the requested resource. Therefore, it's critical that your resources.requests.memory are set realistically (representing baseline usage) to ensure the HPA scales up before the application hits its limits.

1. Ensure Metrics Server is running:

   kubectl get apiservice v1beta1.metrics.k8s.io -o yaml
   

   If it returns Accepted: True, metrics server is likely working.

2. Define HPA: Create a YAML file (e.g., hpa-my-app.yaml):

   apiVersion: autoscaling/v2
   kind: HorizontalPodAutoscaler
   metadata:
     name: my-app-hpa
     namespace: default
   spec:
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment
       name: my-app-deployment # Target your Deployment or StatefulSet
     minReplicas: 1
     maxReplicas: 5
     metrics:
     - type: Resource
       resource:
         name: memory
         target:
           type: Utilization
           averageUtilization: 70 # Scale up when average memory utilization reaches 70% of requested memory
   

   In this example, if resources.requests.memory for my-app-deployment is 512Mi, the HPA will trigger scaling when the average memory utilization across pods reaches 70% of 512Mi (approx. 358Mi).

3. Apply HPA:

   kubectl apply -f hpa-my-app.yaml
   

8. Node OOM Killer Tweaks (Highly Advanced / Last Resort)

For highly specific system-level components where an OOMKilled event on the host node itself would be catastrophic, and cgroup limits aren't sufficient, you might consider adjusting oom_score_adj for critical host processes.

This is generally not recommended for application containers running in Kubernetes. Kubernetes' cgroup management already provides isolation. Manually adjusting oom_score_adj at the host level can interfere with container runtime behavior and potentially destabilize the node or other workloads. Use with extreme caution and only after exhausting all other options.

For a Kubernetes node on Ubuntu 22.04 LTS (which uses cgroup v2), the container runtime (e.g., containerd) manages the memory.max (equivalent to memory.limit_in_bytes in cgroup v1) for containers within their respective cgroups. The kernel's OOM killer respects these cgroup boundaries.

If you absolutely must prevent a host process from being OOMKilled, you could theoretically modify its oom_score_adj. For example, to make kubelet less likely to be OOMKilled (it's usually configured this way by default in production clusters):

# This is an example for a host process, NOT a container process.
# Generally, don't do this for Kubernetes managed applications.
echo -1000 | sudo tee /proc/<PID_of_process>/oom_score_adj

Again, this is a last resort and typically outside the scope of fixing application Pod OOMKilled errors, which are best addressed by adjusting Pod resource limits or application-level memory management.