Troubleshooting NodeJS PM2 Infinite Restart Loop & Memory Leak on Alpine Linux

Resolve NodeJS PM2 infinite restarts and memory leaks on Alpine Linux. Diagnose OOM kills, optimize Node.js, and configure PM2 for stability.

A common and frustrating issue for DevOps engineers managing Node.js applications on Alpine Linux, especially within Docker containers, is an application process managed by PM2 entering an infinite restart loop due to a memory leak. This guide provides a highly technical, step-by-step approach to diagnose and resolve this critical stability problem, ensuring your Node.js services remain performant and resilient.

Symptom & Error Signature

The primary symptom is your Node.js application becoming unresponsive or unavailable, with PM2 repeatedly restarting the service. While PM2 is designed to keep services alive, continuous restarts often mask an underlying, more severe issue – typically an Out Of Memory (OOM) event.

You might observe the following:

• Application status via PM2:

  pm2 status
  

  ┌─────┬───────────┬─────────────┬─────────┬─────────┬──────────┬─────────┬────────┬─────┬───────────┬───────────┐
  │ id  │ name      │ namespace   │ version │ mode    │ pid      │ uptime  │ │ restart │ status  │ cpu       │ memory    │
  ├─────┼───────────┼─────────────┼─────────┼─────────┼──────────┼─────────┼────────┼─────┼───────────┼───────────┤
  │ 0   │ my-app    │ default     │ 1.0.0   │ fork    │ 0        │ 0s      │ 1234   │ errored │ 0%        │ 0B        │
  └─────┴───────────┴─────────────┴─────────┴─────────┴──────────┴─────────┴────────┴─────┴───────────┴───────────┘
  

  The status column showing errored or rapidly cycling through starting and errored, with a very high restart count.

• PM2 application logs:

  pm2 logs my-app --lines 50
  

  You may see messages indicating memory exhaustion within Node.js:

  <--- Last few GCs --->
  [1234:0x5e2b00000000]   200000 ms: Mark-sweep 2047.8 (2051.8) -> 2047.8 (2051.8) MB, 1000.0 / 0.0 ms  (average mu = 0.000) last co
  llection was slow
  [1234:0x5e2b00000000]   200000 ms: Mark-sweep 2047.8 (2051.8) -> 2047.8 (2051.8) MB, 1000.0 / 0.0 ms  (average mu = 0.000) last co
  llection was slow
  
  <--- JS stacktrace --->
  FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
  

  Or simply repetitive restarts without a clear Node.js error:

  0|my-app   | [PM2][WARN] `my-app` failed to start, will restart in 1000ms
  0|my-app   | [PM2][INFO] Starting 'my-app' (0)
  0|my-app   | [PM2][INFO] App 'my-app' started
  0|my-app   | [PM2][ERROR] Script `my-app.js` exited with code 1
  0|my-app   | [PM2][WARN] `my-app` failed to start, will restart in 1000ms
  

• System/Kernel Logs (for OOM Killer):

  dmesg -T | grep -i "oom-killer"
  

  Or (if inside a Docker container using the host's kernel messages) inspect the host's /var/log/messages or journalctl.

  [Thu Oct 29 10:30:00 2026] Out of memory: Kill process 12345 (node) score 999 or sacrifice child
  [Thu Oct 29 10:30:00 2026] Killed process 12345 (node) total-vm:4000000kB, anon-rss:2000000kB, file-rss:0kB, shmem-rss:0kB
  

  This is the definitive sign that the kernel is forcefully terminating your Node.js process due to system-wide memory exhaustion.

• Docker Container Logs (if applicable):

  docker logs <container_name>
  

  You might see Exit Code 137 (SIGKILL, often from OOM) or explicit OOM messages from the Docker daemon.

  ...
  [PM2] Starting /app/server.js in fork_mode (1 instance)
  [PM2] Done.
  [PM2] Logs for my-app
  ...
  0|my-app | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
  ...
  container_name exited with code 137
  

Root Cause Analysis

The core of this problem is almost always a memory leak within the Node.js application itself, exacerbated by the environment.

1. Primary Cause: Node.js Memory Leak:

   • Unmanaged Objects: The application creates objects, data structures, or maintains references that are never properly garbage collected. Common culprits include:
     • Lingering closures or event listeners.
     • Improperly cleared caches.
     • Global variables holding large data.
     • Accumulation of request-specific data without proper cleanup.
     • Stream operations not being properly ended or piped.
   • Heap Exhaustion: Over time, the application's JavaScript heap grows until it hits its configured limit (max_old_space_size). At this point, Node.js attempts garbage collection, but if it cannot free enough memory, it exits with a FATAL ERROR: JavaScript heap out of memory.

2. Secondary Cause: System-Level OOM Killer:

   • When the Node.js process's total memory footprint (RSS – Resident Set Size) exceeds the available physical memory (or cgroup limits in a container) and the system swap space is exhausted (or non-existent, common in Alpine containers), the Linux kernel's Out Of Memory (OOM) killer steps in.
   • The OOM killer selects and terminates processes with a high OOM score to free up memory, leading to the Node.js process being abruptly killed with SIGKILL (exit code 137 in Docker). This kill occurs before Node.js can even log a FATAL ERROR about heap limits.

3. PM2's Role:

   • PM2 is configured by default to restart applications that exit. While beneficial for transient errors, it continuously restarts a leaking application, creating an infinite loop that further strains system resources and makes diagnosis difficult without proper logging.

4. Alpine Linux & Container Considerations:

   • Resource Constraints: Docker containers, especially those built on Alpine, are often provisioned with minimal RAM. This makes them more susceptible to OOM kills than applications running on a beefier host with ample swap.
   • Musl libc vs. Glibc: Alpine uses Musl libc instead of Glibc. While largely compatible, subtle differences in memory allocation behavior or malloc implementations have historically caused some native addons or specific Node.js versions to behave differently, though this is less common with modern Node.js versions.
   • No Swap by Default: Most Docker containers do not have swap configured, meaning memory pressure directly leads to OOM killer invocation rather than slowdowns from swapping.

Step-by-Step Resolution

Resolving this issue requires a systematic approach to diagnose the actual memory usage, prevent premature OOM kills, and ultimately, fix the underlying leak.

1. Initial Diagnosis: Verify OOM Killer & PM2 Status

First, confirm if the OOM killer is the culprit or if Node.js is exiting gracefully with a heap error.

• Check PM2 status and logs:

  pm2 status
  pm2 logs my-app --lines 100
  

  Look for the "FATAL ERROR: JavaScript heap out of memory" message. If you see this, Node.js itself is hitting its heap limit. If you don't see it and only observe restarts, the OOM killer is more likely.

• Check system kernel messages for OOM events:

  dmesg -T | grep -i "oom-killer"
  

  If running in a Docker container, check the host machine's dmesg or journalctl. If you see entries indicating node was killed, the OOM killer is active.

• If using Docker, inspect container logs:

  docker ps -a # Find your container ID or name
  docker inspect <container_id_or_name> | grep "OOMKilled"
  

  A true value for OOMKilled directly confirms the container was terminated by the OOM killer. Also, check docker logs <container_id_or_name> for exit code 137.

2. Analyze Application Memory Usage & Behavior

To prevent the PM2 restart loop from hindering inspection, temporarily adjust PM2's restart behavior.

• Temporarily disable auto-restart (or increase delay): Modify your ecosystem.config.js:

  module.exports = {
    apps : [{
      name   : "my-app",
      script : "./server.js",
      // Other configurations...
      // Temporarily disable restarts for debugging (set to false)
      // or increase delay significantly
      autorestart: false, // Set to false to prevent PM2 from restarting it immediately
      restart_delay: 60000 // Or set a long delay (e.g., 60 seconds)
    }]
  };
  

  Then pm2 reload ecosystem.config.js. Once the application inevitably crashes, it won't immediately restart, giving you a window to inspect.

• Monitor memory usage: Before the crash, use tools like top, htop, or ps to observe the Node.js process's Resident Set Size (RSS).

  # On the host (or inside the container if available)
  watch -n 1 'ps aux | grep node | grep -v grep | awk "{print $2, $4, $6, $11}"'
  # PID %MEM RSS_KB COMMAND
  

  Look for continuous, unbounded growth in the RSS column.

3. Optimize Node.js Memory Allocation & PM2 Configuration

This step helps manage the symptoms while you track down the root cause. It does not fix a memory leak but can provide more stability and better debugging output.

• Increase Node.js V8 Heap Size: If Node.js is hitting its JavaScript heap out of memory error before the OOM killer, you might be under-allocating its heap.

  Increasing max-old-space-size is a temporary measure or a fine-tuning step. It does not fix a memory leak. It merely gives the leaking application more room to consume memory before crashing. If the leak is severe, it will eventually consume this larger space too. The value should be set based on the actual expected peak memory usage, leaving headroom for the OS and other processes. Note that max-old-space-size specifies the limit for the old generation heap, not the total process memory (RSS). The total RSS will always be higher.

  Add NODE_OPTIONS to your PM2 ecosystem file:

  module.exports = {
    apps : [{
      name   : "my-app",
      script : "./server.js",
      exec_mode: "fork", // or "cluster"
      // ... other settings
      env: {
        NODE_ENV: "production",
        NODE_OPTIONS: "--max-old-space-size=2048" // Set to 2GB (in MB)
      },
      // For PM2 v5+, use args to pass Node.js flags directly for easier understanding
      // args: ["--max-old-space-size=2048", "./server.js"],
      // You might need to adjust `script` if using `args` this way.
      // The `env` approach is generally more reliable for Node.js options with PM2.
    }]
  };
  

  After modifying, run pm2 reload my-app or pm2 restart ecosystem.config.js.

• Configure PM2 max_memory_restart: PM2 can be configured to restart an application if its memory usage exceeds a defined threshold. This is a "soft" restart, triggered by PM2, not the kernel's OOM killer.

  Set max_memory_restart to a value lower than the system's OOM killer threshold (e.g., container memory limit minus a buffer) and lower than the max-old-space-size if you've set it high. This allows PM2 to gracefully restart the app before the kernel ruthlessly kills it, preventing SIGKILL and giving you cleaner logs.

  module.exports = {
    apps : [{
      name   : "my-app",
      script : "./server.js",
      // ... other settings
      max_memory_restart: "1800M", // Restart if RSS exceeds 1.8GB. Adjust as needed.
      // It's crucial this is less than your container's memory limit.
    }]
  };
  

  Reload PM2: pm2 reload ecosystem.config.js.

• Docker Container Memory Limits: If your application is containerized, ensure Docker provides sufficient memory.

  docker run -d --name my-app-container -p 80:3000 -m 2g --memory-swap -1 my-node-app:latest
  

  • -m 2g: Sets the memory limit to 2GB for the container.
  • --memory-swap -1: Allows the container to use as much swap space as available on the host (or no swap if -m is set and --memory-swap isn't specified, which is common). Consider setting it explicitly to 0 if you want no swap in the container and rely solely on RAM. For Alpine in particular, often no swap is configured.

4. Identify and Fix Memory Leaks in Application Code

This is the most challenging but ultimately most crucial step. You need to profile your Node.js application to find where memory is being consumed and not released.

• Enable Node.js Inspector: Run your application with the inspector enabled. This allows you to connect Chrome DevTools for heap snapshots and CPU profiling.

  // ecosystem.config.js
  module.exports = {
    apps : [{
      name   : "my-app",
      script : "./server.js",
      exec_mode: "fork",
      env: {
        NODE_ENV: "production",
        NODE_OPTIONS: "--inspect=0.0.0.0:9229 --max-old-space-size=2048" // Listen on all interfaces
      }
    }]
  };
  

  Reload PM2. Then, port forward or expose port 9229 to your local machine.

  # If running in Docker
  docker run -p 9229:9229 ...
  # If using SSH on a remote server
  ssh -L 9229:localhost:9229 user@your-server-ip
  

  Open chrome://inspect in your browser. Click "Open dedicated DevTools for Node" next to your application's target.

• Take Heap Snapshots with Chrome DevTools:

  1. In DevTools, go to the "Memory" tab.
  2. Select "Heap Snapshot".
  3. Click "Take snapshot".
  4. Interact with your application (trigger actions that you suspect might cause a leak).
  5. Take a second snapshot after a period of activity.
  6. Compare the two snapshots. Look for objects that are growing in count and retained size that should have been garbage collected. Filter by "Comparison" to see new objects.

  Focus on the "Retainers" section for identified leaking objects. This shows the path from the global object to the leaking object, helping you pinpoint the code responsible for holding onto it.

• Code Review for Common Leak Patterns:

  • Event Emitters/Listeners: Ensure event listeners are properly removed (.off() or .removeListener()) when they are no longer needed, especially in long-lived processes or when components are destroyed.
  • Closures: Be wary of closures that capture variables from outer scopes, preventing them from being garbage collected if the closure itself is long-lived.
  • Caches: Implement robust cache eviction policies (LRU, TTL) for in-memory caches. Without limits, caches become memory sinks.
  • Timers/Intervals: Clear setInterval or setTimeout calls with clearInterval or clearTimeout when they are no longer necessary.
  • Streams: Ensure all readable and writable streams are properly ended or destroyed to release underlying resources.
  • Global Variables: Avoid storing large, mutable objects in global variables unless strictly necessary and managed carefully.

5. PM2 Configuration Best Practices

Once the leak is addressed, refine your PM2 configuration for optimal stability and monitoring.

• Re-enable autorestart and sensible restart_delay:

  module.exports = {
    apps : [{
      name   : "my-app",
      script : "./server.js",
      // ...
      autorestart: true,
      restart_delay: 1000 // 1 second, typical
    }]
  };
  

• Set kill_timeout and listen_timeout: These ensure PM2 gives your application time to shut down gracefully before sending SIGKILL.

  module.exports = {
    apps : [{
      name   : "my-app",
      script : "./server.js",
      // ...
      kill_timeout: 5000, // Give 5 seconds for graceful shutdown
      listen_timeout: 8000 // If app doesn't emit 'ready' in 8s, assume problem
    }]
  };
  

• Use PM2 Cluster Mode (if applicable): For CPU-bound tasks, cluster mode can utilize multiple CPU cores. For memory leaks, it will launch multiple instances, each potentially leaking, so fix the leak first. However, it does provide process isolation, and a crash in one instance doesn't bring down the whole application.

  module.exports = {
    apps : [{
      name   : "my-app",
      script : "./server.js",
      instances: "max", // Or a specific number like 4
      exec_mode: "cluster",
      // ...
    }]
  };
  

• Centralized Logging: Consider forwarding PM2 logs to a centralized logging system (e.g., ELK stack, Grafana Loki, CloudWatch Logs) for easier monitoring and historical analysis.

By systematically diagnosing OOM events, carefully profiling your Node.js application for memory leaks, and applying robust PM2 configurations, you can achieve a stable and performant Node.js environment even on resource-constrained Alpine Linux setups.