Troubleshooting NodeJS PM2 Infinite Restart Loop & Memory Leak on Debian 12 Bookworm

Fix PM2 application crashes due to NodeJS memory leaks on Debian 12. Diagnose and resolve service instability with expert-level steps.

Introduction

Experiencing a NodeJS application constantly restarting, showing an "errored" or "restarting" status in PM2, and your website becoming unresponsive or serving HTTP 50x errors? This common problem, especially prevalent in long-running Node.js processes, often points to a memory leak. On Debian 12 Bookworm, with its modern kernel and systemd integration, an unaddressed memory leak will quickly lead to the Linux OOM (Out Of Memory) Killer terminating your NodeJS process, causing PM2 to cycle endlessly in an attempt to restore service. This guide provides a highly technical, step-by-step approach to diagnose and resolve such issues.

Symptom & Error Signature

Users will typically observe their web application becoming slow, unresponsive, or returning 502 Bad Gateway (if proxied by Nginx) or 503 Service Unavailable errors.

From the server perspective, you'll see:

1. PM2 Status:

   pm2 list
   

   ┌────┬────────────────────┬───────────┬──────┬─────────┬─────────┬───────────┬────────┬─────────────────┬───────────┐
   │ id │ name               │ namespace │ version │ mode    │ pid     │ uptime    │ ↺    │ status          │ cpu       │
   ├────┼────────────────────┼───────────┼──────┼─────────┼─────────┼───────────┼────────┼─────────────────┼───────────┤
   │ 0  │ my-node-app        │ default   │ 1.0.0 │ fork    │ 23456   │ 0s        │ 127    │ errored         │ 0%        │
   └────┴────────────────────┴───────────┴──────┴─────────┴─────────┴───────────┴────────┴─────────────────┴───────────┘
   

   Notice the errored status and the high ↺ (restart count). The uptime might also frequently reset to 0s.

2. PM2 Logs:

   pm2 logs my-node-app --lines 100
   

   You'll frequently encounter entries similar to these, indicating a memory allocation failure:

   0|my-node-app  | <--- Last few GCs --->
   0|my-node-app  |
   0|my-node-app  | [23456:0x5113000]    42045 ms: Scavenge 2046.2 (2055.9) -> 2038.3 (2056.9) MB, 5.0 / 0.0 ms  (average mu = 0.983, current mu = 0.963) allocation failure
   0|my-node-app  | [23456:0x5113000]    42054 ms: Scavenge 2046.8 (2056.9) -> 2039.0 (2057.9) MB, 4.0 / 0.0 ms  (average mu = 0.983, current mu = 0.963) allocation failure
   0|my-node-app  | [23456:0x5113000]    42065 ms: Scavenge 2047.5 (2057.9) -> 2039.8 (2058.9) MB, 5.2 / 0.0 ms  (average mu = 0.983, current mu = 0.963) allocation failure
   0|my-node-app  |
   0|my-node-app  | <--- JS stacktrace --->
   0|my-node-app  |
   0|my-node-app  | Fatal process OOM in OldSpace
   0|my-node-app  |
   0|my-node-app  | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
   0|my-node-app  |
   0|my-node-app  | 1: 0x5608d1f7c0c8 node::Abort() [node]
   0|my-node-app  | 2: 0x5608d1f7c1d0 node::FatalError(char const*, char const*) [node]
   0|my-node-app  | 3: 0x5608d211e40a v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*) [node]
   # ... more stack trace ...
   0|my-node-app  | PM2 | App name: 'my-node-app' id:0 online
   0|my-node-app  | PM2 | Process my-node-app killed by PM2
   0|my-node-app  | PM2 | Starting my-node-app in fork_mode (fork)
   0|my-node-app  | PM2 | App name: 'my-node-app' id:0 online
   # ... and the cycle repeats ...
   

3. System Logs (OOM Killer):

   journalctl -k -p err | grep -i "oom-killer"
   # or
   dmesg -T | grep -i "out of memory"
   

   Output indicating the Linux kernel's Out-Of-Memory killer intervened:

   [Mon Oct 17 08:30:15 2026] node invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, req_pid=23456, req_uid=1001
   [Mon Oct 17 08:30:15 2026] Out of memory: Killed process 23456 (node) total-vm:4234567kB, anon-rss:2048765kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:4567kB oom_score_adj:0
   [Mon Oct 17 08:30:15 2026] oom_reaper: reaped process 23456 (node), now reaping 12 bytes of memory
   

Root Cause Analysis

The core of the problem is almost always a memory leak within the NodeJS application itself. PM2, a process manager, simply restarts the application when it crashes, masking the underlying issue. The infinite restart loop is PM2 doing its job, but without addressing the leak, the process quickly exhausts available memory again.

Underlying reasons for memory leaks in NodeJS often include:

• Unreleased References: Objects or variables are inadvertently kept in memory by other parts of the application, preventing the JavaScript garbage collector from reclaiming them. Common culprits include:
  • Closures: Functions retaining references to their outer scope variables even when no longer needed.
  • Event Emitters: Listeners registered with EventEmitter.on() that are never explicitly removed with EventEmitter.off() or allowed to be garbage collected.
  • Global Variables/Caches: Growing arrays or objects stored in the global scope (e.g., global, process.env, or simple module-scoped variables) that continuously accumulate data without a cleanup mechanism or size limit.
• Excessive Data Caching: Implementing caching logic without proper eviction policies (e.g., LRU – Least Recently Used) can lead to caches growing indefinitely.
• Large Objects in Scope: Holding onto very large data structures (e.g., database query results, large file buffers) longer than necessary within a function's scope.
• Third-Party Library Issues: Bugs or inefficient memory usage in external NPM packages.
• Stream Handling: Improperly handling Node.js streams (e.g., not draining readable streams, not ending writable streams, or backpressure issues) can lead to accumulating buffers.
• Recursive Functions: Deep or infinite recursion can lead to stack overflow, but also memory exhaustion if objects are created at each step without being released.
• Debugging Tools/Middleware: Sometimes, development or debugging tools or middleware in a production environment can inadvertently hold onto references.

When the NodeJS process's memory footprint grows beyond its allocated V8 heap limit, it triggers a "JavaScript heap out of memory" error. If it exceeds system-wide or cgroup-defined memory limits, the Linux kernel's OOM Killer steps in, forcefully terminating the process to maintain system stability.

Step-by-Step Resolution

Solving a memory leak requires a combination of mitigation, diagnosis, and code-level fixes.

1. Confirm PM2 Status and Initial Logs

Before diving deep, verify the observed symptoms.

# Check overall PM2 status
pm2 list

# View logs for your specific application
pm2 logs my-node-app --lines 200

# Check system journal for OOM Killer messages
sudo journalctl -k -p err | grep -i "oom-killer"
sudo dmesg -T | grep -i "out of memory"

2. Implement PM2 Memory Limit (Temporary Mitigation)

PM2 can be configured to restart your application when it hits a defined memory threshold. This prevents the OOM Killer from unilaterally terminating your process and provides a cleaner restart cycle. This is a mitigation, not a fix for the underlying leak.

You can set this in your ecosystem.config.js:

// ecosystem.config.js
module.exports = {
  apps: [
    {
      name: "my-node-app",
      script: "index.js",
      instances: "max", // Or a specific number of instances
      exec_mode: "cluster", // Or "fork"
      watch: false,
      max_memory_restart: "500M", // Restart if memory usage exceeds 500MB
      env: {
        NODE_ENV: "production",
      },
    },
  ],
};

Apply the changes:

pm2 reload ecosystem.config.js --env production
pm2 save

Choose max_memory_restart value carefully. Set it below the total available RAM and below the point where your application starts showing "JavaScript heap out of memory" errors but above its normal steady-state memory consumption. This allows PM2 to gracefully restart before system-level OOM issues.

3. Monitor Memory Usage

Continuous monitoring is crucial to understand the growth pattern.

• PM2 Monitor:

  pm2 monit
  

  This provides a real-time terminal dashboard of CPU, memory, and logs for your PM2 processes.

• System-wide Monitoring:

  htop
  free -h
  

  htop offers an interactive, colorful view of processes. free -h shows overall memory usage. Look for a node process with a steadily increasing RES (Resident Set Size) memory.

• Detailed Process Memory:

  pmap -x <PID_of_node_process>
  

  This command shows a detailed memory map for a specific process, useful for advanced analysis.

4. Diagnose NodeJS Memory Leak with Heap Snapshots

This is the most critical step for identifying the source of the leak within your code.

Prerequisites:

• Install heapdump module:

  npm install heapdump
  

• Enable gc for inspection (optional, but recommended for debugging): Start your Node.js application with the --expose-gc flag. This allows you to manually trigger garbage collection, which can be useful when comparing heap snapshots.

  // In your ecosystem.config.js or via pm2 start command
  module.exports = {
    apps: [
      {
        name: "my-node-app",
        script: "index.js",
        node_args: "--expose-gc --inspect=0.0.0.0:9229", // Enable GC and remote debugger
        // ... other configs
      },
    ],
  };
  

  Reload PM2 to apply these arguments.

  pm2 reload ecosystem.config.js
  

Steps for Heap Snapshot Analysis:

1. Modify your application code to generate heap snapshots: Insert heapdump.writeSnapshot() calls at strategic points or based on certain conditions (e.g., after processing a request, after a certain time interval).

   // In your application's main file (e.g., index.js)
   const heapdump = require('heapdump');
   const path = require('path');
   
   // Make sure to create a directory for snapshots, e.g., /tmp/heapdumps
   const SNAPSHOT_DIR = '/tmp/heapdumps';
   if (!require('fs').existsSync(SNAPSHOT_DIR)) {
     require('fs').mkdirSync(SNAPSHOT_DIR);
   }
   
   let snapshotCounter = 0;
   setInterval(() => {
     const filename = path.join(SNAPSHOT_DIR, `heapdump-${Date.now()}-${snapshotCounter++}.heapsnapshot`);
     console.log(`Generating heap snapshot: ${filename}`);
     heapdump.writeSnapshot(filename);
     // Optional: Manually trigger GC if --expose-gc is enabled
     // if (global.gc) {
     //   global.gc();
     //   console.log('Manually triggered garbage collection.');
     // }
   }, 5 * 60 * 1000); // Generate every 5 minutes
   

   Generating heap snapshots is resource-intensive and can temporarily pause your Node.js process. Do this on a staging environment or during low-traffic periods in production. Remove the heapdump code and --expose-gc flag once debugging is complete.

2. Collect multiple heap snapshots: Allow your application to run for some time, ideally long enough for the memory to visibly increase. Collect at least two snapshots, spaced apart (e.g., 30 minutes to an hour), to observe memory growth.

3. Analyze snapshots using Chrome DevTools:

   • Open Chrome browser, then open DevTools (F12).
   • Go to the "Memory" tab.
   • Click the "Load" button and load your .heapsnapshot files one by one.
   • Select "Comparison" from the dropdown next to the second snapshot. This shows the delta between the two snapshots.
   • Sort by "Size Delta" or "Objects Delta" to identify objects that have significantly increased in count or size. This is usually where your leak resides. Look for custom objects, large arrays, or specific data structures growing without bounds.

4. Alternative: Node.js Diagnostics Report: For a less intrusive approach, you can use Node.js's built-in diagnostic reports. Set environment variable NODE_REPORT_SIGNAL=SIGUSR2 (or another signal) and send that signal to your Node.js process:

   # In ecosystem.config.js:
   env: {
     NODE_REPORT_SIGNAL: "SIGUSR2"
   }
   # Then trigger report:
   kill -SIGUSR2 <PID_of_node_process>
   

   This generates a JSON report with heap stats, CPU usage, and more. While not as detailed as a heap snapshot for specific object analysis, it can point to general memory issues.

5. Analyze Codebase for Common Leak Patterns

Based on your heap snapshot analysis, examine your code for:

• Unclosed Event Listeners:

  const EventEmitter = require('events');
  const myEmitter = new EventEmitter();
  
  function processRequest(req, res) {
    // If listener is added here for every request and never removed
    // it will accumulate.
    myEmitter.on('data', () => { /* ... */ });
    // SHOULD HAVE: myEmitter.once('data', ...) or myEmitter.off('data', ...)
  }
  

• Growing Global Arrays/Objects:

  const cache = {}; // Global or module-scoped cache
  function addData(key, value) {
    cache[key] = value; // If 'key' is dynamic and never removed, this leaks.
  }
  

  Consider using an LRU cache (e.g., lru-cache npm module) with a max and ttl (time-to-live) for bounded caching.
• Long-lived Closures: Functions that retain references to large objects from their outer scope.
• Unsettled Promises/Timeouts: Promises that never resolve/reject, or setTimeout/setInterval calls that are never cleared, can hold references.
• Stream Issues: Ensure all streams are properly piped, error-handled, and closed.

  const fs = require('fs');
  function readFileAndProcess(filePath, res) {
      const stream = fs.createReadStream(filePath);
      stream.on('error', (err) => {
          console.error('Stream error:', err);
          stream.destroy(); // Important to destroy on error
          res.status(500).send('Error');
      });
      // If the stream is not consumed or piped to somewhere, it could buffer indefinitely
      stream.pipe(res); // Properly pipes and handles backpressure
  }
  

6. Optimize PM2 Configuration (Advanced)

Beyond max_memory_restart, consider:

• Clustering: Use instances: "max" with exec_mode: "cluster". This utilizes all CPU cores and, if one instance leaks, only that instance might restart, not the entire application. However, if the leak is systemic, all instances will eventually restart.
• --kill-timeout: Increase this if your app needs more time to gracefully shut down.
• --stop-exit-codes: If your app exits with specific codes for specific reasons, PM2 can handle them differently.

pm2 start ecosystem.config.js --env production --update-env
pm2 save

7. Update NodeJS and PM2

Newer versions of Node.js often include performance improvements, better garbage collection, and bug fixes related to memory management. Similarly, PM2 updates can offer better stability and features.

• NodeJS Update (using n or nvm):

  # Using nvm (Node Version Manager)
  nvm install --lts
  nvm use --lts
  nvm alias default --lts # Set LTS as default
  # Or using 'n' module
  sudo npm install -g n
  sudo n lts
  

  Then restart PM2 to pick up the new Node.js version:

  pm2 delete all
  pm2 startup # Re-create systemd service if necessary
  pm2 start ecosystem.config.js
  pm2 save
  

• PM2 Update:

  npm install -g pm2@latest
  pm2 update # Update PM2 in-place (often requires a restart of the daemon)
  

Always test major updates in a staging environment before deploying to production. Backward compatibility is generally good, but breaking changes can occur.

8. Containerization with Docker (Isolation)

While Docker doesn't fix the memory leak, it provides excellent resource isolation. If your NodeJS app is containerized, a leak will only affect that container and not the entire host system or other services.

Example Dockerfile:

FROM node:18-bookworm-slim
WORKDIR /app
COPY package*.json ./
RUN npm install --production
COPY . .
CMD ["pm2-runtime", "ecosystem.config.js"] # Using pm2-runtime for Docker

Run with memory limits:

docker build -t my-node-app .
docker run -d --name my-leaky-app 
  --memory="512m" --memory-swap="1g"  # Limit RAM and swap usage
  --restart always  # Docker will restart the container if it crashes
  my-node-app

Use docker stats to monitor memory usage of containers.

Docker's resource limits (via cgroups) will act similarly to the OOM killer, but confined to the container. The --restart always policy mirrors PM2's restart behavior. This is a robust way to prevent system-wide instability.

9. System-Level Resource Limits (Systemd/cgroups – Last Resort)

For a non-containerized setup, you can enforce hard memory limits on the PM2 service itself using systemd cgroup directives. This will ensure that the Node.js process gets killed by the kernel when it hits the limit, preventing it from consuming all system resources, but it doesn't solve the leak.

1. Identify PM2's systemd service: Typically, PM2 runs as a user service. You can find its configuration using:

   pm2 startup
   # Look for output like: '[PM2] To setup the Startup Script under your user account run: sudo env PATH=$PATH:/usr/bin /usr/local/bin/pm2 startup systemd -u <user> --hp /home/<user>'
   # The actual service file might be in ~/.config/systemd/user/pm2-<user>.service
   

   If it's a global service, it might be in /etc/systemd/system/pm2-<user>.service.

2. Edit the systemd service file:

   # For a user service:
   systemctl --user edit pm2-<user>.service
   # For a global service:
   sudo systemctl edit pm2-<user>.service
   

   Add or modify the [Service] section:

   [Service]
   MemoryMax=768M # Set a hard memory limit, e.g., 768MB
   # You can also use MemoryHigh= to define a threshold for soft reclaim
   # MemoryAccounting=yes # Ensure memory accounting is enabled
   

   Save and exit.

3. Reload systemd and restart PM2:

   # For user service:
   systemctl --user daemon-reload
   systemctl --user restart pm2-<user>.service
   # For global service:
   sudo systemctl daemon-reload
   sudo systemctl restart pm2-<user>.service
   

Setting MemoryMax will cause the kernel's OOM Killer to terminate your Node.js process once this limit is hit, resulting in PM2 restarts. This is a last resort to protect the system from a runaway process, but it is not a solution to the underlying memory leak. Only use this if you've exhausted other options and need immediate system stability.