Node.js PM2 Service Restarts: Troubleshooting Memory Leaks on Ubuntu 22.04 LTS

Debug and resolve Node.js PM2 processes stuck in an infinite restart loop due to memory leaks on Ubuntu 22.04. Learn to identify, diagnose, and fix the root cause.

A Node.js application deployed with PM2 is a common and robust setup for production environments on Ubuntu servers. However, encountering a scenario where your PM2-managed Node.js service continuously restarts in an infinite loop, often indicating a memory leak, can be a frustrating and critical issue. This guide will provide a highly technical, step-by-step approach to diagnose and resolve such memory leak-induced restart loops on Ubuntu 22.04 LTS, restoring stability to your web applications.

Symptom & Error Signature

The primary symptom is an application that becomes unresponsive, serves HTTP 502 Bad Gateway errors (if proxied by Nginx), or fails to start correctly. Upon inspection, you'll observe the PM2 process repeatedly restarting, consuming increasing amounts of memory before crashing.

Typical error signatures visible in your logs might include:

• PM2 logs (pm2 logs <app_name>): You'll see repetitive entries indicating process exits and restarts.

  PM2 | App [my-node-app:0] starting in development mode...
  PM2 | App [my-node-app:0] online
  PM2 | App [my-node-app:0] exited with code [0] via signal [SIGINT]
  PM2 | App [my-node-app:0] is restarting
  PM2 | App [my-node-app:0] starting in development mode...
  PM2 | App [my-node-app:0] online
  # ... this cycle repeats
  

  Or, if max_memory_restart is configured in PM2:

  PM2 | App [my-node-app:0] reached max memory restart threshold (200MB)
  PM2 | App [my-node-app:0] exited with code [0] via signal [SIGINT]
  PM2 | App [my-node-app:0] is restarting
  

• System logs (journalctl -xe or dmesg -T | grep -i "oom"): The Linux Out Of Memory (OOM) killer might be terminating your process if it exhausts system memory before PM2's max_memory_restart kicks in.

  # journalctl -xe | grep -i "oom"
  kernel: Out of memory: Killed process 12345 (node) total-vm:1024MB, anon-rss:900MB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:2000kB oom_score_adj:0
  kernel: oom_reaper: reaped process 12345 (node), now dead
  

• Process monitoring (htop or top): Observing your Node.js process consuming steadily increasing RAM until it crashes or is restarted.

Root Cause Analysis

The fundamental cause of this issue is a memory leak within your Node.js application, exacerbated by PM2's automatic restart capabilities and potentially the Linux kernel's OOM killer.

1. Application-Level Memory Leak: This is the most common and significant root cause. A memory leak occurs when your Node.js application continuously allocates memory without releasing it, leading to a gradual increase in RAM usage over time. Common culprits include:

   • Unreferenced Objects: Objects held in closures, global variables, or long-lived caches that are no longer needed but prevent the garbage collector from reclaiming their memory.
   • Unclosed Resources: Unhandled stream errors, database connection pools not properly released, or file descriptors left open.
   • Event Listener Accumulation: Adding event listeners repeatedly without removing them can lead to a build-up of event handlers, each holding a reference to its scope.
   • Growing Data Structures: Arrays, maps, or sets that continuously accumulate data without a clear eviction strategy.
   • Third-party Modules: Bugs or memory mismanagement within a dependency.

2. PM2 max_memory_restart Threshold: PM2 offers a protective max_memory_restart option. When the application's memory usage exceeds this threshold, PM2 gracefully restarts the process. While intended to prevent system-wide memory exhaustion, if a leak is present, this merely creates an infinite restart loop without addressing the underlying problem.

3. Linux Kernel OOM Killer: If the memory leak is severe and your Node.js process consumes all available system memory (or reaches a critical threshold before PM2's max_memory_restart can act), the Linux kernel's OOM (Out Of Memory) killer will step in. It terminates processes it deems "memory hogs" to preserve system stability, often selecting the Node.js application, which PM2 then immediately attempts to restart.

Step-by-Step Resolution

Addressing a memory leak requires a methodical approach, combining monitoring, debugging, and code-level fixes.

1. Verify and Monitor Current State

Before attempting fixes, confirm the symptoms and gather initial data.

# Display PM2 process list and status
pm2 list

# Monitor real-time PM2 logs for your application
# Replace 'my-node-app' with your application's name or ID
pm2 logs my-node-app --lines 200

# Monitor system-wide resource usage (watch for Node.js process memory)
htop

# Check system memory
free -h

# Check kernel messages for OOM killer activity
dmesg -T | grep -i "oom-killer" | tail -n 10

2. Analyze PM2 Configuration

Examine your PM2 ecosystem file (ecosystem.config.js or json) for relevant settings like max_memory_restart.

// ecosystem.config.js example
module.exports = {
  apps: [
    {
      name: "my-node-app",
      script: "src/index.js",
      instances: "max",
      exec_mode: "cluster",
      watch: false,
      max_memory_restart: "250M", // Adjust this value cautiously
      env: {
        NODE_ENV: "production",
      },
      log_file: "/var/log/pm2/my-node-app.log",
      error_file: "/var/log/pm2/my-node-app-error.log",
      out_file: "/var/log/pm2/my-node-app-out.log",
      merge_logs: true,
    },
  ],
};

A max_memory_restart value too low can exacerbate the restart loop. While debugging, you might temporarily increase it to allow more time for profiling, but this is not a solution to the leak. A very high value could lead to OOM killer activation.

3. Temporarily Disable or Adjust max_memory_restart for Debugging

To allow your application to run longer for profiling, you might temporarily disable or increase max_memory_restart (e.g., to 0 or a very large number, like 2G, if you have sufficient RAM).

# To update an existing app:
pm2 stop my-node-app
pm2 set my-node-app max_memory_restart 0  # Disables restart on memory threshold
# Or to increase it: pm2 set my-node-app max_memory_restart 1024M
pm2 start my-node-app

# If using an ecosystem file, update the file and then:
pm2 reload ecosystem.config.js

Disabling max_memory_restart is for debugging purposes only and can lead to complete system memory exhaustion if the leak is severe, potentially causing critical services to crash or the server to become unresponsive due to the OOM killer. Re-enable or set an appropriate threshold once debugging is complete.

4. Configure OOMScoreAdjust for Critical Services (Mitigation)

To prevent the OOM killer from prematurely terminating essential services (like PM2 itself or your database) while debugging, you can adjust their OOMScoreAdjust via systemd. A lower score means it's less likely to be killed.

# Example for PM2 systemd service. First, find its service file:
systemctl status pm2-$(whoami).service

# Typically located at /etc/systemd/system/pm2-$(whoami).service
# Edit the service file (use sudo with your preferred editor, e.g., nano or vim)
sudo nano /etc/systemd/system/pm2-$(whoami).service

Add OOMScoreAdjust=-500 (or similar negative value) under the [Service] section:

[Unit]
Description=PM2 process manager
After=network.target

[Service]
Type=forking
User=your_user
Group=your_user
ExecStart=/usr/bin/pm2 resurrect
ExecStop=/usr/bin/pm2 kill
# ... other settings ...
OOMScoreAdjust=-500 # Add this line
Restart=on-failure

[Install]
WantedBy=multi-user.target

# Reload systemd daemon and restart PM2 service
sudo systemctl daemon-reload
sudo systemctl restart pm2-$(whoami).service

5. Debug Node.js Memory Leaks with Profiling Tools

This is the most critical step to identify the source of the leak within your application code.

Option A: Node.js Inspector with Chrome DevTools

1. Start Node.js in inspect mode: You'll need to modify how PM2 launches your app. First, stop your application:

   pm2 stop my-node-app
   

   Then, modify your ecosystem.config.js to include node_args:

   module.exports = {
     apps: [
       {
         name: "my-node-app",
         script: "src/index.js",
         // ...
         node_args: "--inspect=0.0.0.0:9229", // Expose inspector on all interfaces on port 9229
       },
     ],
   };
   

   Or, if starting manually without PM2 for direct debugging (less ideal for production, but good for isolated testing):

   node --inspect=0.0.0.0:9229 src/index.js
   

   Exposing the inspector port (9229) on 0.0.0.0 makes it accessible from anywhere. For production, ensure firewall rules are in place to only allow access from trusted IPs (e.g., your local machine's IP) or tunnel the connection. sudo ufw allow from YOUR_LOCAL_IP to any port 9229

2. Start/Reload PM2 app:

   pm2 reload ecosystem.config.js
   # Or start if it was stopped
   # pm2 start ecosystem.config.js
   

   PM2 logs will show the inspector URL: PM2 | my-node-app-0 | Debugger listening on ws://0.0.0.0:9229/...

3. Connect Chrome DevTools:

   • Open Google Chrome on your local machine.
   • Navigate to chrome://inspect.
   • Click "Configure…" under "Remote Target" and add your server's IP address and port (e.g., your_server_ip:9229).
   • A "Remote Target" entry for your Node.js app should appear. Click "inspect".
   • In the DevTools window, go to the "Memory" tab.

4. Take Heap Snapshots:

   • Click the "Take snapshot" button.
   • Interact with your application (trigger typical usage patterns that might cause the leak).
   • Wait a few minutes (or until memory usage increases noticeably in htop).
   • Take another heap snapshot.
   • Compare the snapshots (Comparison view) to identify objects that are increasing in count or size. Look for constructor names from your code or libraries that show significant growth.

Option B: clinic.js (Recommended for comprehensive analysis)

clinic.js is a powerful suite of tools for Node.js performance and memory profiling. clinic doctor and clinic bubbleprof are particularly useful for memory leaks.

1. Install clinic.js (globally or locally):

   npm install -g clinic
   # Or in your project: npm install --save-dev clinic
   

2. Run your application with clinic doctor: This needs to be run directly on the application, not through PM2 initially.

   # Ensure you are in your application's root directory
   clinic doctor -- node src/index.js
   

   clinic doctor will run your application, collect data, and then generate an HTML report in a new folder (e.g., 12345.clinic).

3. Analyze the Report:

   • Transfer the generated HTML report to your local machine (scp or rsync).
   • Open the HTML file in your browser.
   • Look for the "Memory" section. clinic doctor will highlight memory usage patterns. If it detects a leak, it will often point to the "Top leaking functions" or "Top leaking lines of code".

Option C: heapdump (for programmatic snapshots)

For cases where you can't expose the inspector port or need automated snapshots.

1. Install heapdump:

   npm install --save heapdump
   

2. Integrate into your code: Add code to take heap snapshots periodically or on demand.

   // In your main application file (e.g., src/index.js)
   if (process.env.NODE_ENV === 'development' || process.env.ENABLE_HEAPDUMP) {
     const heapdump = require('heapdump');
     const path = require('path');
     const fs = require('fs');
   
     let snapshotCounter = 0;
     setInterval(() => {
       snapshotCounter++;
       const filename = `heapdump-${Date.now()}-${snapshotCounter}.heapsnapshot`;
       const filepath = path.join(process.cwd(), filename);
       heapdump.writeSnapshot(filepath, (err) => {
         if (err) console.error('Heapdump error:', err);
         else console.log(`Heapdump written to ${filepath}`);
       });
     }, 5 * 60 * 1000); // Take a snapshot every 5 minutes
   }
   

3. Deploy and Collect: Deploy this change, let it run, and collect the .heapsnapshot files from your server.

4. Analyze: Load these .heapsnapshot files into Chrome DevTools (Memory tab -> Load button) for analysis, similar to Option A.

6. Fix the Memory Leak in Your Code

Once the profiling tools have pointed you towards the problematic areas, it's time to modify your application code. Common fixes include:

• Nullifying References: Explicitly set variables to null when their objects are no longer needed, especially in long-running processes or global scopes.
• Removing Event Listeners: Always use removeListener or off when event emitters are no longer needed to prevent listener accumulation.
• Limiting Cache Size: Implement an LRU (Least Recently Used) cache strategy or set maximum sizes for in-memory caches.
• Proper Stream Handling: Ensure all Node.js streams are correctly ended or destroyed, especially on errors.
• Database/Connection Pool Management: Verify that database connections are properly released back to the pool after use.
• Avoid Global State Accumulation: Be mindful of adding data to global arrays or objects in server-side applications, as they persist across requests.

7. Update Node.js and Dependencies

Sometimes, memory leaks can be due to bugs in older Node.js versions or outdated third-party libraries.

1. Update Node.js: Use nvm (Node Version Manager) for easy management.

   # Install nvm if not already present
   curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
   source ~/.bashrc # Or restart terminal
   nvm install 20 # Install latest LTS (or a specific version)
   nvm use 20
   nvm alias default 20 # Set as default
   

2. Update npm/yarn dependencies:

   cd /path/to/your/app
   npm outdated # Check for outdated packages
   npm update   # Update all packages to their latest compatible versions
   # Or if using yarn:
   yarn outdated
   yarn upgrade
   

8. Reconfigure PM2 and Redeploy

After fixing the leak in your code and updating dependencies:

1. Re-enable/Adjust max_memory_restart: Set an appropriate max_memory_restart in your ecosystem.config.js. This acts as a safety net.

   max_memory_restart: "500M", // Example, adjust based on your app's stable memory usage
   

2. Deploy the updated code:

   git pull origin main # Or whatever your deployment strategy is
   npm install          # Install any new or updated dependencies
   pm2 reload ecosystem.config.js --update-env # Reload and update environment variables
   pm2 save             # Save the current process list to be reloaded on boot
   

3. Monitor Closely: Continuously monitor your application's memory usage with pm2 monit and htop to ensure the leak is resolved and the memory profile remains stable over time.

   pm2 monit
   

By systematically identifying, debugging, and resolving the underlying memory leak, you can achieve a stable and performant Node.js application managed by PM2 on your Ubuntu 22.04 LTS server.