Resolve Low System Performance & Swap Thrashing on Ubuntu 20.04 LTS

Optimize Linux swap configuration on Ubuntu 20.04 LTS to mitigate performance degradation, memory exhaustion, and swap thrashing for enhanced system stability.

Introduction

Experiencing sluggish server performance, unresponsive applications, or persistent high load averages on your Ubuntu 20.04 LTS system? Often, these symptoms point to inefficient memory management, specifically related to the Linux kernel's swap space configuration. When physical RAM is exhausted, the operating system moves less frequently used data to a designated area on the hard disk, known as swap space. While this prevents out-of-memory (OOM) errors, excessive swapping (often termed "swap thrashing") can severely degrade system performance due to the drastic speed difference between RAM and disk I/O. This guide provides a comprehensive, expert-level approach to diagnose and resolve low system performance caused by suboptimal swap configurations.

Symptom & Error Signature

Users typically observe applications freezing, slow response times for web services (e.g., Nginx serving pages), database queries timing out, or SSH sessions becoming unresponsive. These issues are often accompanied by specific metrics or log entries:

1. High I/O Wait and Swap Activity (observed via top, htop, vmstat):

# top - 14:30:05 up 1 day, 2:15,  1 user,  load average: 8.50, 7.80, 6.20
# Tasks: 210 total,   1 running, 209 sleeping,   0 stopped,   0 zombie
# %Cpu(s):  5.2 us,  2.1 sy,  0.0 ni, 12.3 id, 80.4 wa,  0.0 hi,  0.0 si,  0.0 st
# MiB Mem :   7987.0 total,   256.0 free,   7200.0 used,    531.0 buff/cache
# MiB Swap:   2047.0 total,     0.0 free,   2047.0 used.    487.0 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM    TIME+ COMMAND
   1234 www-data  20   0 1800.0m 800.0m  50.0m S  10.0  10.0   0:45.12 php-fpm
   5678 mysql     20   0 2.500g 1.500g  100.0m S   5.0  18.0   1:10.20 mysqld
   ...

# vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  2 2096120 200000 150000 300000 1200 800  50000 30000 4000 8000 10  5 10 75  0
 0  1 2096120 180000 140000 280000 1000 700  45000 28000 3800 7500  8  4 12 76  0

• load average: Significantly higher than the number of CPU cores.
• %Cpu(s) ... wa: High wa (I/O wait) percentage, indicating the CPU is idle waiting for disk operations.
• MiB Swap: ... used: Swap space is completely or almost completely utilized.
• vmstat si and so: High values for si (swap in) and so (swap out), indicating active paging between RAM and disk.

2. Out-Of-Memory (OOM) Killer Events (observed via dmesg or syslog):

# dmesg | grep -i oom
[ 1234.567890] mysqld invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, req_zone=DMA32, some_things_dirty=0
[ 1234.567900] Call Trace:
[ 1234.567910]  dump_stack+0x6d/0x8b
[ 1234.567920]  oom_kill_process+0x13c/0x1e0
[ 1234.567930]  out_of_memory+0x211/0x3e0
[ 1234.567940]  __alloc_pages_nodemask+0xab9/0xc20
[ 1234.567950]  alloc_pages_current+0x9e/0x110
[ 1234.567960]  __do_page_cache_readahead+0x10e/0x1f0
[ 1234.567970]  filemap_fault+0x43d/0x790
[ 1234.567980]  __do_fault+0x4c/0xc0
[ 1234.567990]  handle_mm_fault+0x10d2/0x12c0
[ 1234.568000]  __do_page_fault+0x223/0x4e0
[ 1234.568010]  do_page_fault+0x2a/0xe0
[ 1234.568020]  page_fault+0x21/0x30
[ 1234.568030] Killed process 5678 (mysqld) total-vm:7987216kB, anon-rss:1536000kB, file-rss:0kB, shmem-rss:0kB

• The Linux OOM killer is invoked when the system runs out of memory and cannot allocate more for a process, even with swap fully utilized. This results in critical processes being arbitrarily terminated.

Root Cause Analysis

Low system performance due to swap issues typically stems from one or a combination of the following:

1. Insufficient Physical RAM: The most straightforward cause. The applications and services running on the server simply demand more memory than is physically available.
2. Aggressive Swapping (swappiness too high): The swappiness kernel parameter (default typically 60 on Ubuntu) dictates how readily the kernel moves application data from RAM to swap space. A high value means the kernel prefers to swap out even relatively active processes and keep disk caches in RAM, leading to unnecessary I/O on busy servers.
3. Inadequate or Missing Swap Space: If the system has no swap partition/file, or if it's too small, the OOM killer will be invoked much sooner when RAM is depleted.
4. Memory Leaks in Applications: A misbehaving application continuously consumes more memory without releasing it, eventually exhausting all available RAM and swap.
5. vfs_cache_pressure Misconfiguration: This kernel parameter controls the kernel's tendency to reclaim memory used for directory and inode caches. A high value (default 100) means the kernel aggressively reclaims this memory, potentially leading to increased disk I/O if the caches are frequently needed.
6. Slow Storage for Swap: If swap space resides on a slow HDD or a highly saturated storage array, even legitimate swapping can bring the system to a crawl.

Step-by-Step Resolution

#### 1. Assess Current Memory and Swap Usage

Before making any changes, understand your current system's memory profile.

# Display human-readable memory and swap usage
free -h

# Interactive process viewer, showing memory, swap, and CPU usage
# Look for processes with high %MEM and high SWAP usage (if visible)
htop

# Report virtual memory statistics. 'si' (swap in) and 'so' (swap out) are key.
# Run with 'vmstat 1' to see real-time updates.
vmstat 1 5

Example free -h output indicating high swap usage:

              total        used        free      shared  buff/cache   available
Mem:          7.8Gi       7.5Gi       150Mi        10Mi       200Mi       100Mi
Swap:         2.0Gi       2.0Gi          0B

#### 2. Analyze Application Memory Consumption

Identify which processes are the heaviest memory consumers. This helps in understanding if specific applications are misbehaving or if the workload simply exceeds available RAM.

# List processes by Resident Set Size (RSS), which is actual physical memory used
ps aux --sort -rss | head -n 10

# More detailed memory usage per process and user (install 'smem' if not present)
# sudo apt install smem
smem -k -H -P "php-fpm|nginx|mysqld" # Adjust regex for your critical services

#### 3. Adjust swappiness Parameter

swappiness ranges from 0 to 100. A high value (default 60) makes the kernel eager to swap. For servers, especially those running databases or memory-intensive applications, a lower swappiness value is often beneficial, instructing the kernel to keep more application data in RAM.

The optimal swappiness value is workload-dependent. Common server recommendations range from 10 to 30. A value of 0 on older kernels (pre-5.0) could disable swapping almost entirely and risk OOM killer. On modern kernels (Ubuntu 20.04 runs 5.x), swappiness=0 means "swap only when absolutely necessary to avoid OOM," but it's generally safer to use a low non-zero value like 10.

a. Check current swappiness value:

cat /proc/sys/vm/swappiness

b. Change swappiness temporarily (until next reboot):

sudo sysctl vm.swappiness=10

c. Make swappiness change persistent:

Edit /etc/sysctl.conf and add or modify the following line:

sudo nano /etc/sysctl.conf

Add this line to the file:

vm.swappiness=10

Replace 10 with your desired value. For databases, some administrators even use 1. Experiment and monitor.

d. Apply changes without reboot:

sudo sysctl -p

#### 4. Configure vfs_cache_pressure

vfs_cache_pressure controls the kernel's tendency to reclaim memory that is used for caching directory and inode objects. The default value is 100. A lower value makes the kernel less aggressive in reclaiming this memory, which can improve performance for workloads that frequently access the filesystem metadata.

a. Check current vfs_cache_pressure value:

cat /proc/sys/vm/vfs_cache_pressure

b. Change vfs_cache_pressure temporarily:

sudo sysctl vm.vfs_cache_pressure=50

c. Make vfs_cache_pressure change persistent:

Edit /etc/sysctl.conf and add or modify the following line:

sudo nano /etc/sysctl.conf

Add this line to the file:

vm.vfs_cache_pressure=50

A value of 50 or 20 is often a good starting point for servers. Too low might mean less memory available for other uses, so balance is key.

d. Apply changes without reboot:

sudo sysctl -p

#### 5. Create or Resize Swap Space

If your system lacks swap, or if the existing swap is too small (e.g., less than 1-2x RAM, or less than 4GB for systems with 4-8GB RAM), consider adding more. A swap file is generally easier to manage and resize than a swap partition.

Ensure the disk where you create the swap file has sufficient free space and reasonable I/O performance. Creating swap on a slow or already bottlenecked disk can worsen performance.

a. Check for existing swap space:

sudo swapon --show
sudo free -h

If swapon --show returns nothing, you have no active swap.

b. Create a new swap file (e.g., 4GB):

Replace 4G with your desired size. The recommended size often depends on RAM, but typically 1-2x RAM up to 4GB-8GB is sufficient for most server workloads. For systems with a lot of RAM (e.g., 32GB+), 4-8GB of swap is often enough as a buffer.

# Create a 4GB file named swapfile in the root directory
sudo fallocate -l 4G /swapfile

# Set restrictive permissions for the swap file
sudo chmod 600 /swapfile

# Mark the file as swap space
sudo mkswap /swapfile

# Enable the swap file
sudo swapon /swapfile

c. Verify the new swap space:

sudo swapon --show
sudo free -h

d. Make the swap file persistent across reboots:

Edit the /etc/fstab file:

sudo nano /etc/fstab

Add the following line to the end of the file:

/swapfile none swap sw 0 0

Verify the path to your swap file is correct if you chose a different name or location. Incorrect entries in /etc/fstab can prevent your system from booting. Test sudo mount -a after editing fstab to check for syntax errors.

#### 6. Monitor System Performance Post-Resolution

After applying these changes, it's crucial to monitor your system closely to ensure the adjustments have had the desired effect.

# Real-time process and resource monitoring
htop

# Virtual memory statistics, observe 'si' and 'so'
vmstat 1

# Disk I/O statistics (install 'sysstat' if not present)
# sudo apt install sysstat
iostat -x 1

Look for decreased wa (I/O wait), lower si/so values, and improved overall system responsiveness. If performance improves, you've likely identified and mitigated the swap-related bottleneck.

#### 7. Identify and Mitigate Memory Leaks (If Applicable)

If performance issues persist even after optimizing swap, or if the OOM killer continues to terminate applications, a memory leak might be the underlying problem.

• Application-Specific Tools: Many programming languages and frameworks offer memory profiling tools (e.g., valgrind for C/C++, pprof for Go, tracemalloc for Python).
• Log Analysis: Review application logs for errors related to memory allocation or resource exhaustion.
• Developer Collaboration: Work with developers to analyze application code and identify potential leaks.

#### 8. Consider Hardware Upgrade

Ultimately, if your workload consistently demands more RAM than your server provides, and all software optimizations have been exhausted, the most effective solution is a hardware upgrade—specifically, adding more physical RAM. Swap space is a fallback, not a replacement, for sufficient physical memory.