Troubleshooting Redis OOM: ‘Command Not Allowed When Used Memory Limit Reached’ on Ubuntu 22.04 LTS

Resolve the critical 'Redis OOM command not allowed' error on Ubuntu 22.04 LTS. Diagnose memory limits, optimize Redis configuration, and restore application functionality swiftly.

When your application relies on Redis for caching, session management, or real-time data, encountering the "OOM command not allowed" error can bring critical functionality to a halt. This issue signifies that your Redis instance has hit its configured memory limit and is actively refusing write operations, leading to application errors, slow performance, or outright service interruptions. This guide provides a comprehensive, expert-level approach to diagnosing and resolving this common Redis memory constraint on an Ubuntu 22.04 LTS server.

Symptom & Error Signature

Users will typically experience degraded application performance, data not being saved, or specific application features failing. On the server side, you will observe the following error messages in your application logs (e.g., PHP-FPM logs, Node.js console, Python traceback) and within the Redis server logs.

Application Log Example (e.g., PHP-FPM/Laravel):

[2023-10-27 10:30:15] production.ERROR: RedisException: OOM command not allowed when used memory > 'maxmemory'. in /var/www/html/vendor/predis/predis/src/Client.php:370
Stack trace:
#0 /var/www/html/vendor/predis/predis/src/Client.php(370): PredisClient->onErrorResponse(Object(PredisResponseError))
#1 /var/www/html/vendor/predis/predis/src/Client.php(339): PredisClient->executeCommand(Object(PredisCommandStringSet))
#2 /var/www/html/vendor/laravel/framework/src/Illuminate/Redis/Connections/PhpRedisConnection.php(102): PredisClient->__call('set', Array)
#3 ...

Redis Server Log Example (/var/log/redis/redis-server.log or journalctl -u redis):

27 Oct 2023 10:29:58.123 # WARNING: OOM command not allowed. Used memory > maxmemory. Redis will still accept read-only commands.

When checking Redis directly using redis-cli, you might also see the maxmemory_reached flag.

redis-cli info memory | grep -E "used_memory_human|maxmemory_human|maxmemory_policy|maxmemory_reached"

Expected output showing the issue:

used_memory_human:1.00G
maxmemory_human:1.00G
maxmemory_policy:noeviction
maxmemory_reached:1

Root Cause Analysis

The "OOM command not allowed" error occurs when the Redis instance's memory usage (used_memory) exceeds its configured maxmemory limit, and the maxmemory-policy is set to noeviction or a similar policy that prevents Redis from automatically removing keys to free up space.

Here's a breakdown of the underlying reasons:

1. maxmemory Limit Reached: The most direct cause. Redis is configured with a maxmemory directive in its redis.conf file, which specifies the maximum amount of RAM it is allowed to consume. Once this limit is hit, Redis's behavior is dictated by its maxmemory-policy.
2. maxmemory-policy: noeviction: This is often the default or a common setting. When noeviction is active, Redis will return an error on write commands (like SET, LPUSH, INCR) if maxmemory is reached. It will still allow read-only commands. Other eviction policies (like allkeys-lru, volatile-ttl) attempt to free up memory by deleting keys, which can lead to data loss.
3. Increased Data Volume: Your application is storing more data in Redis than anticipated, or the dataset has grown over time without corresponding scaling of the Redis instance.
4. Inefficient Data Structures or Large Keys:
   • Storing very large strings, lists, or hashes that consume significant memory per key.
   • Using Redis data structures inefficiently (e.g., storing JSON objects as large strings instead of using Hashes for individual fields).
   • Lack of proper key expiry (EXPIRE commands not being used or TTLs being too long).
5. Memory Fragmentation: Redis allocates memory from the operating system. Over time, as keys are added, updated, and deleted, the memory can become fragmented, meaning Redis might be holding onto more physical RAM than its reported used_memory suggests, or it might struggle to find contiguous blocks for new data. A high mem_fragmentation_ratio (> 1.5) indicates this.
6. Background Saving (RDB/AOF): When Redis performs a background save (either RDB snapshot or AOF rewrite), it uses a "copy-on-write" mechanism. This means that if data is modified during the save operation, Redis needs to duplicate those memory pages. This can temporarily increase the overall memory footprint significantly, potentially pushing it over the maxmemory limit.
7. Other System Processes: While less common for OOM command not allowed, if the host system is critically low on RAM, the OS might swap Redis data to disk, causing performance issues, or in extreme cases, trigger the OS's OOM killer, terminating Redis itself.

Step-by-Step Resolution

Addressing this error requires a methodical approach, balancing immediate relief with long-term stability and performance.

1. Assess Current Redis and System Memory Status

Before making any changes, gather crucial data about your Redis instance and the host system.

# Check Redis memory details
redis-cli info memory

# Output will include details like:
# used_memory:1073741824
# used_memory_human:1.00G
# used_memory_rss:1100000000
# used_memory_peak:1100000000
# used_memory_peak_human:1.02G
# used_memory_overhead:10000000
# used_memory_startup:1000000
# used_memory_dataset:1063741824
# used_memory_dataset_perc:99.00%
# total_system_memory:2097152000
# total_system_memory_human:2.00G
# maxmemory:1073741824
# maxmemory_human:1.00G
# maxmemory_policy:noeviction
# mem_fragmentation_ratio:1.10
# mem_allocator:jemalloc-5.1.0

# Check Redis configuration for maxmemory and policy
grep -E "maxmemory|maxmemory-policy" /etc/redis/redis.conf

# Check overall system memory
free -h

# Check Redis logs for recent OOM warnings
sudo journalctl -u redis-server -n 100 --no-pager | grep "OOM command not allowed"

Analyze the redis-cli info memory output, paying close attention to used_memory_human, maxmemory_human, maxmemory_policy, and mem_fragmentation_ratio. Compare Redis's memory usage with free -h to see how much RAM is available on the server.

2. Adjust Redis maxmemory Limit (Increase if System Allows)

The most direct solution is to increase the maxmemory limit, provided your server has sufficient free RAM.

Do NOT set maxmemory to a value close to or exceeding the total physical RAM of your server. This can lead to system-wide out-of-memory issues, OS swapping, and potentially crash the entire server. Always leave sufficient RAM for the OS and other critical services (Nginx, PHP-FPM, database, etc.). A good rule of thumb is to allocate no more than 60-80% of total physical RAM to Redis, depending on other services.

1. Edit the Redis configuration file:

   sudo nano /etc/redis/redis.conf
   

2. Locate the maxmemory directive. It might be commented out by default. Uncomment it (remove #) and set a new, higher value. Use suffixes like g for gigabytes or m for megabytes.

   # Example: Increase maxmemory to 2GB
   maxmemory 2gb
   

3. Save the file (Ctrl+X, Y, Enter).

4. Restart the Redis service for the changes to take effect:

   sudo systemctl restart redis-server
   

5. Verify the new maxmemory:

   redis-cli info memory | grep maxmemory_human
   

3. Choose an Appropriate maxmemory-policy

If you cannot increase maxmemory or if Redis is primarily used as a cache, changing the eviction policy can prevent the OOM error by allowing Redis to automatically free up space.

Changing the maxmemory-policy from noeviction to an eviction policy means Redis will automatically delete keys when the maxmemory limit is reached. This can lead to data loss if your application relies on all data being persistently available in Redis. Understand the implications for your specific use case.

1. Edit the Redis configuration file:

   sudo nano /etc/redis/redis.conf
   

2. Locate the maxmemory-policy directive.

3. Choose a suitable policy:

   • noeviction: (Current policy, returns error on writes).
   • allkeys-lru: Evicts keys that were least recently used, regardless of TTL. Best for generic caching where some data loss is acceptable.
   • volatile-lru: Evicts least recently used keys that have an expire set.
   • allkeys-random: Evicts random keys.
   • volatile-random: Evicts random keys that have an expire set.
   • volatile-ttl: Evicts keys with the shortest remaining TTL that have an expire set.
   • allkeys-lfu: Evicts keys that were least frequently used (Redis 4.0+).
   • volatile-lfu: Evicts least frequently used keys that have an expire set (Redis 4.0+).

   For most caching scenarios, allkeys-lru is a common and effective choice.

   # Example: Set policy to allkeys-lru
   maxmemory-policy allkeys-lru
   

4. Save the file and restart Redis:

   sudo systemctl restart redis-server
   

4. Optimize Redis Data Structures and Application Usage

This is a long-term solution that requires application-level changes but is crucial for sustainable performance.

1. Identify Large Keys: Use the redis-cli --bigkeys command to find keys that consume a lot of memory.

   redis-cli --bigkeys
   

   Analyze the output and determine if these large keys are necessary or can be broken down.

2. Implement Key Expiry (TTL): Ensure that temporary data in Redis (e.g., sessions, temporary caches) has a TTL (Time To Live) set using EXPIRE or SETEX. This allows Redis to automatically remove stale data.

3. Use Efficient Data Structures:

   • Instead of storing large JSON strings, consider using Redis Hashes for structured objects.
   • Use Sorted Sets for leaderboards or ordered data.
   • Explore specialized structures like HyperLogLogs for unique counting or Bitmaps for boolean flags if appropriate.

4. Avoid Proliferating Keys: Clean up unused keys. Regularly audit your key space.

5. Address Memory Fragmentation

A high mem_fragmentation_ratio can silently consume more memory. Redis 4.0+ offers active defragmentation.

1. Check Fragmentation Ratio:

   redis-cli info memory | grep mem_fragmentation_ratio
   

   If this value is consistently above 1.5 (or even 1.0 if used_memory_rss is significantly higher than used_memory), active defragmentation can help.

2. Enable Active Defragmentation (Redis 4.0+): Edit /etc/redis/redis.conf and ensure these lines are uncommented and set:

   activedefrag yes
   active-defrag-ignore-bytes 100mb
   active-defrag-threshold-lower 10
   active-defrag-threshold-upper 100
   active-defrag-cycle-min 5
   active-defrag-cycle-max 75
   

   These settings tell Redis to actively defragment memory when certain conditions are met, consuming some CPU cycles but reducing fragmentation.

3. Restart Redis after making these changes.

6. Scale Your Redis Instance

If optimization doesn't suffice or your data volume consistently grows, scaling is necessary.

1. Upgrade Server RAM: The simplest form of scaling is to provision a server with more physical RAM and then adjust Redis's maxmemory accordingly (refer back to Step 2).

2. Implement Redis Clustering: For very large datasets or high traffic, a Redis Cluster distributes data across multiple nodes, effectively sharding your data and increasing overall memory capacity and throughput. This is a significant architectural change.

3. Dedicated Redis Server: If Redis is running on a shared server with other resource-intensive applications, consider moving it to a dedicated VM or physical server to isolate its resources.

4. Utilize External Managed Redis Services: Cloud providers like AWS ElastiCache, Azure Cache for Redis, or Google Cloud Memorystore offer managed Redis services that handle scaling, high availability, and operational overhead.

7. Implement Robust Monitoring

Ongoing monitoring is essential to prevent future OOM issues and understand Redis's health.

• Redis Metrics: Monitor used_memory, used_memory_rss, maxmemory, keyspace_hits, keyspace_misses, mem_fragmentation_ratio.
• System Metrics: Keep an eye on overall server RAM usage, swap usage, and CPU load.
• Alerting: Set up alerts when Redis memory usage approaches maxmemory or when mem_fragmentation_ratio becomes high.

Tools like Prometheus + Grafana, Datadog, or New Relic can provide excellent visibility into Redis performance and system health.