Troubleshooting ‘Redis connection refused cluster node down’ on Ubuntu 20.04 LTS

Resolve 'Redis connection refused cluster node down' on Ubuntu 20.04. A comprehensive guide to diagnose and fix Redis cluster connectivity and configuration issues.

A "Redis connection refused" error, especially when compounded with "cluster node down" or "failure config," is a critical issue indicating that your application cannot establish a connection to your Redis cluster. This typically results in service interruptions, degraded performance, or complete outages for applications relying on Redis for caching, session management, or real-time data. On an Ubuntu 20.04 LTS system, this can stem from a myriad of underlying causes, from networking misconfigurations to a truly unhealthy Redis cluster state.

This comprehensive guide, crafted from 16 years of web hosting and DevOps experience, will walk you through diagnosing and resolving this complex issue, ensuring your Redis cluster returns to an optimal, highly available state.

Symptom & Error Signature

Users will typically experience application failures, slow loading times, or complete inaccessibility of features relying on Redis. You'll observe specific error messages in application logs, Redis server logs, or when attempting to interact with Redis via redis-cli.

Common Application Log Errors (Examples for PHP, Node.js, Python):

[2026-11-13 10:30:01] production.ERROR: RedisException: Connection refused [tcp://127.0.0.1:6379] in /var/www/html/app/Http/Controllers/MyController.php:45
Stack trace:
#0 /var/www/html/vendor/predis/predis/src/Connection/StreamConnection.php(141): PredisConnectionStreamConnection->connect()
#1 /var/www/html/vendor/predis/predis/src/Connection/AbstractConnection.php(95): PredisConnectionStreamConnection->connect()
#2 /var/www/html/vendor/predis/predis/src/Client.php(329): PredisConnectionAbstractConnection->connect()

Error: Redis connection to 10.0.0.5:6379 failed - connect ECONNREFUSED
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1146:16) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '10.0.0.5',
  port: 6379
}

redis.exceptions.ConnectionError: Error 111 connecting to 10.0.0.6:6379. Connection refused.

Redis Server Log Errors (from journalctl -xeu redis-server or /var/log/redis/redis-server.log):

12345:M 13 Nov 2026 10:30:01.234 # OOM encountered and system is not allowing memory allocations.
12345:M 13 Nov 2026 10:30:02.567 # Can't open the log file: Permission denied
12345:M 13 Nov 2026 10:30:03.890 # Fatal error: Can't initialize cluster node configuration, please check 'cluster-config-file' and ensure it's writable.
12345:M 13 Nov 2026 10:30:04.123 # Cluster state is not UP, failing client commands.

redis-cli Output:

redis-cli -h <down_node_ip> -p 6379
Could not connect to Redis at <down_node_ip>:6379: Connection refused

When checking cluster status from a healthy node:

redis-cli -c -p 6379 cluster nodes
# Output might show one or more nodes in a 'fail' state:
c1f2e3d4a5b6c7d8e9f0a1b2c3d4e5f6g7h8i9j0 10.0.0.5:6379@16379 master - 0 1678888888888 1 connected
k1l2m3n4o5p6q7r8s9t0u1v2w3x4y5z6a7b8c9d0 10.0.0.6:6379@16379 master,fail 0 1678888888888 2 disconnected

Root Cause Analysis

The "connection refused" error typically means the client attempted to connect to a Redis instance, but no process was listening on the specified IP address and port, or a firewall explicitly blocked the connection. When combined with "cluster node down" or "failure config," the potential root causes expand:

1. Redis Server is Down or Crashed:

   • The redis-server process might not be running on the affected node(s).
   • It could have crashed due to an Out-Of-Memory (OOM) error, segmentation fault, or other critical system issues.
   • Incorrect systemd unit file configuration preventing proper startup.

2. Network Connectivity Issues:

   • Firewall: ufw on Ubuntu, cloud provider security groups (AWS Security Groups, Azure Network Security Groups, Google Cloud Firewall Rules) blocking port 6379 (and 16379 for cluster bus communication).
   • Incorrect bind directive: Redis is configured to listen only on 127.0.0.1 or a specific internal IP, but the application/other cluster nodes are trying to connect from a different IP.
   • Network Partition: The specific node might be isolated from the rest of the cluster or the application server due to routing problems or subnet issues.

3. Redis Configuration Errors (redis.conf):

   • protected-mode yes: If bind is 127.0.0.1 (or not set) and protected-mode is yes, Redis will only accept connections from localhost.
   • cluster-enabled no: The node is not configured to be part of a cluster.
   • cluster-config-file issues: The specified nodes.conf file is missing, unreadable, or unwritable, preventing the cluster from forming or persisting its state.
   • Incorrect cluster-announce-ip / cluster-announce-port: Nodes advertise incorrect IP/port combinations to other cluster members, leading to communication failures.

4. Cluster State Inconsistencies (nodes.conf):

   • The nodes.conf file on one or more nodes might be corrupted or out of sync, leading to a fractured view of the cluster.
   • A node was forcefully removed or improperly re-added, causing the cluster to mark it as fail.
   • Split-brain scenarios where nodes have divergent cluster configurations.

5. Resource Exhaustion:

   • Memory: Redis consuming too much RAM, leading to OOM killer termination by the kernel.
   • Open File Descriptors: Redis hitting ulimit restrictions for open files, preventing new connections.
   • Disk Space: Insufficient disk space for AOF or RDB persistence files, leading to write errors or server halts.

6. Application Configuration Mismatch:

   • The application is trying to connect to a non-existent or incorrect Redis IP/port.
   • The application client library is not configured for Redis Cluster mode and is attempting to connect to a single node endpoint expecting a standalone instance.

Step-by-Step Resolution

Follow these steps meticulously to identify and resolve the issue. Prioritize checking the most common causes first.

1. Verify Redis Service Status

The first step is to ensure the Redis server process is actually running on the affected node(s).

1. Check Service Status:

   sudo systemctl status redis-server
   

   Look for "Active: active (running)". If it's not running, or shows "failed", proceed to check logs.

2. Examine Redis Logs for Errors:

   sudo journalctl -xeu redis-server --since "1 hour ago"
   

   Alternatively, check the Redis log file, typically located at /var/log/redis/redis-server.log (as configured in redis.conf):

   sudo tail -f /var/log/redis/redis-server.log
   

   Look for OOM, segmentation fault, Permission denied, Fatal error, or specific CLUSTERDOWN messages indicating why the service might have failed to start or crashed.

3. Attempt to Start/Restart Redis:

   sudo systemctl start redis-server
   # Or, if already running but unstable:
   sudo systemctl restart redis-server
   

   After restarting, check systemctl status redis-server again and re-examine logs for new errors.

2. Check Network Connectivity and Firewall

A "connection refused" often points to a network block or an incorrect bind address.

1. Test Basic Network Connectivity: From the application server (or another cluster node), ping the affected Redis node's IP:

   ping <redis_node_ip>
   

   If ping fails, you have a fundamental network problem (routing, network interface down).

2. Test Port Reachability with telnet or nc: From the application server (or another cluster node), attempt to connect to the Redis port (default 6379) and the cluster bus port (default 16379):

   telnet <redis_node_ip> 6379
   telnet <redis_node_ip> 16379
   # If telnet is not installed:
   nc -vz <redis_node_ip> 6379
   nc -vz <redis_node_ip> 16379
   

   If telnet or nc shows "Connection refused" or hangs, a firewall or bind issue is likely. If it connects successfully, the Redis process is listening.

3. Inspect Local Firewall (UFW) on Redis Node:

   sudo ufw status verbose
   

   Ensure that port 6379/tcp and 16379/tcp (for cluster bus) are allowed from the IPs of your application servers and other Redis cluster nodes. If not, add them:

   sudo ufw allow 6379/tcp comment 'Allow Redis clients'
   sudo ufw allow 16379/tcp comment 'Allow Redis cluster bus'
   sudo ufw reload
   

   Allowing access to Redis from 0.0.0.0/0 (all IPs) is a significant security risk. Always restrict access to known, trusted IP addresses or subnets.

4. Check Cloud Provider Security Groups/Firewall Rules: If your Redis nodes are in a cloud environment (AWS EC2, Azure VMs, Google Cloud Compute Engine), verify that the relevant security groups or network firewall rules permit inbound connections on ports 6379 and 16379 from your application servers and other cluster nodes.

3. Inspect Redis Configuration (redis.conf)

The primary Redis configuration file is typically at /etc/redis/redis.conf.

1. Open redis.conf:

   sudo nano /etc/redis/redis.conf
   

2. Review bind Directive:

   • Problem: If bind 127.0.0.1 is set, Redis will only listen on the localhost interface. Other cluster nodes or remote applications won't be able to connect.
   • Solution: Change bind to the specific IP address(es) Redis should listen on, or to 0.0.0.0 to listen on all available network interfaces.

     # Bind to a specific IP address (recommended for security)
     bind 10.0.0.5
     
     # OR bind to all interfaces (less secure, use with strong firewalls)
     # bind 0.0.0.0
     

   After changing bind, you must restart the Redis service: sudo systemctl restart redis-server.

3. Review protected-mode:

   • Problem: If protected-mode yes is combined with bind 127.0.0.1 (or no bind directive), only localhost connections are allowed.
   • Solution: If bind is set to a public/internal IP, you can keep protected-mode yes. If you need to bind to 0.0.0.0 for some reason, consider changing protected-mode no ONLY if your firewall rules are extremely strict and you understand the security implications.

     protected-mode no
     

   Disabling protected-mode without proper firewall rules exposes your Redis instance to the internet, making it vulnerable to unauthorized access and attacks. Use with extreme caution.

4. Verify Cluster Directives: Ensure these are correctly set for a cluster node:

   cluster-enabled yes
   cluster-config-file nodes.conf
   # This path is relative to the working directory. Often /var/lib/redis/
   # For Ubuntu, it's typically /var/lib/redis/nodes.conf
   cluster-node-timeout 15000 # Adjust as needed, default is 15 seconds
   

   Also, check cluster-announce-ip and cluster-announce-port if you're behind NAT or Docker/containerized environments where the perceived IP/port might differ from the actual one.

   # If behind NAT or using Docker, uncomment and set the publicly reachable IP/port
   # cluster-announce-ip <public_ip_of_node>
   # cluster-announce-port 6379
   # cluster-announce-bus-port 16379
   

   After any configuration changes, save the file and restart Redis: sudo systemctl restart redis-server.

4. Examine Redis Cluster State

Assuming Redis is now running and reachable, check its cluster health.

1. Connect to a Healthy Node and Check Cluster Info: From any healthy node in your cluster, use redis-cli in cluster mode:

   redis-cli -c -h <healthy_node_ip> -p 6379 cluster info
   

   Look for cluster_state: ok. If it's fail, something is seriously wrong. Also check cluster_known_nodes to see if all expected nodes are listed.

2. Check Individual Node Status:

   redis-cli -c -h <healthy_node_ip> -p 6379 cluster nodes
   

   Examine the output. You're looking for nodes marked fail or PFAIL (possible fail) and their IP addresses. If the problematic node is listed as fail or disconnected, it confirms the cluster knows about the issue.

3. Attempt to Reset Cluster State (Cautious Approach): If a node is truly down or has a corrupted nodes.conf, you might need to reset its cluster state.

   CLUSTER RESET HARD will delete all data on the node if it's a master and force it to forget its cluster configuration. Only use this if you are absolutely sure the node has no critical data or if you are deliberately re-initializing a cluster node. CLUSTER RESET SOFT is safer as it only resets the configuration but keeps the data.

   On the problematic node itself:

   # Stop Redis first
   sudo systemctl stop redis-server
   
   # Make a backup of nodes.conf
   sudo cp /var/lib/redis/nodes.conf /var/lib/redis/nodes.conf.bak.$(date +%F_%H%M%S)
   
   # Use redis-cli to perform the reset (needs Redis to be running, so start it first, then connect)
   # Alternatively, you can just delete nodes.conf and let Redis recreate it on startup for a fresh start.
   # sudo rm /var/lib/redis/nodes.conf
   
   # Start Redis
   sudo systemctl start redis-server
   
   # Then connect to it and perform the reset if necessary
   redis-cli -h 127.0.0.1 -p 6379 CLUSTER RESET SOFT
   # OR (DANGER: DATA LOSS POSSIBLE if master)
   # redis-cli -h 127.0.0.1 -p 6379 CLUSTER RESET HARD
   
   # After reset, the node needs to be re-added to the cluster (if it was a master/replica)
   # This involves using redis-cli --cluster add-node or --cluster replicate
   

   After a RESET SOFT or HARD and nodes.conf recreation, the node will restart as a standalone instance. You'll need to use redis-cli --cluster to either add it as a new master or a replica to an existing master:

   # Example: Add as a new master to an existing cluster
   redis-cli --cluster add-node <new_node_ip>:6379 <existing_node_ip>:6379
   
   # Example: Add as a replica to an existing master
   redis-cli --cluster add-node <replica_node_ip>:6379 <master_node_ip>:6379 --cluster-slave
   

5. Check nodes.conf File

The nodes.conf file (typically /var/lib/redis/nodes.conf on Ubuntu) contains the cluster state for each node. Corruption or inconsistencies here can lead to nodes being marked down.

1. Inspect the file:

   sudo cat /var/lib/redis/nodes.conf
   

   Look for unusual entries, repeated node IDs, or invalid IP addresses. This file is managed by Redis and should not be manually edited under normal circumstances.

2. Back up and Delete (Advanced/Last Resort): If you suspect severe corruption and CLUSTER RESET doesn't help, and you are certain the node has no critical data (e.g., it's a new node or a replica that can be resynced), you can:

   sudo systemctl stop redis-server
   sudo cp /var/lib/redis/nodes.conf /var/lib/redis/nodes.conf.bak.$(date +%F_%H%M%S)
   sudo rm /var/lib/redis/nodes.conf
   sudo systemctl start redis-server
   

   Then, the node will start as a fresh, unassigned cluster member, and you'll need to re-add it to the cluster using redis-cli --cluster add-node as described in Step 4.

6. Application Configuration Validation

Ensure your application is configured correctly to connect to your Redis Cluster.

1. Verify Redis Host/Port in Application Config: Check your application's environment variables, .env file, or configuration files (e.g., config/database.php for Laravel, settings.py for Django, config.js for Node.js). Ensure the IP address and port used match a healthy node in your Redis cluster. For cluster-aware clients, you usually provide a seed list of nodes.

2. Use a Cluster-Aware Client Library: For Redis Cluster, your application must use a client library that supports Redis Cluster mode (e.g., phpredis, go-redis, redis-py with RedisCluster, ioredis). These clients handle automatic slot remapping and failover, connecting to any node in the cluster and being redirected as needed. A non-cluster-aware client will often try to connect to a single node and fail if that node isn't responsible for the requested key or goes down.

7. System Resource Monitoring

Resource exhaustion can cause Redis to crash or become unresponsive, leading to "connection refused."

1. Check Memory Usage:

   free -h
   

   Look for low available memory. If Redis is using a large percentage, it might be susceptible to the Out-Of-Memory (OOM) killer. Check dmesg for OOM events:

   dmesg | grep -i oom
   

   If OOM errors appear, you need to either reduce Redis memory usage (maxmemory directive in redis.conf), add more RAM, or scale down your data.

2. Check Open File Descriptors: Redis uses file descriptors for connections and persistence. A low ulimit can cause issues.

   ulimit -n
   

   The output should be a reasonably high number (e.g., 10000 or more). If it's low (e.g., 1024), you might need to increase it in /etc/security/limits.conf and in your systemd unit file for Redis. Edit /etc/security/limits.conf:

   redis soft nofile 65536
   redis hard nofile 65536
   

   Edit /etc/systemd/system/redis-server.service (or override it):

   [Service]
   # ...
   LimitNOFILE=65536
   

   After changes, run sudo systemctl daemon-reload and sudo systemctl restart redis-server.

3. Check Disk Space:

   df -h
   

   If your Redis is configured for AOF or RDB persistence, ensure there's enough free disk space in the data directory (typically /var/lib/redis). Full disks can prevent Redis from writing persistence files, potentially causing it to halt.

8. Advanced Cluster Troubleshooting (Rejoining/Recreating)

If all simpler steps fail, and your cluster is severely fragmented or broken beyond repair via individual node fixes, you might need more drastic measures.

1. Fix Cluster Command: Redis provides a command to attempt to fix a broken cluster.

   redis-cli --cluster fix <any_node_ip>:6379
   

   This command can resolve common cluster inconsistencies, but may not work for all scenarios.

2. Recreating the Cluster (Last Resort, Data Loss Risk): If the cluster is unrecoverable, and you have recent backups, or the data is transient/can be rebuilt, the most drastic step is to destroy and recreate the cluster.

   This action will result in complete data loss unless you have a robust backup and restore strategy. Proceed with extreme caution and only as a last resort.

   • Stop Redis on ALL nodes:

     sudo systemctl stop redis-server
     

   • Delete nodes.conf and dump.rdb/AOF files on ALL nodes:

     sudo rm /var/lib/redis/nodes.conf
     sudo rm /var/lib/redis/dump.rdb # If RDB persistence is enabled
     sudo rm /var/lib/redis/*.aof # If AOF persistence is enabled
     

   • Start Redis on ALL nodes:

     sudo systemctl start redis-server
     

     Each node will now start as a standalone Redis instance.
   • Create the new cluster:

     redis-cli --cluster create <node1_ip>:6379 <node2_ip>:6379 <node3_ip>:6379 ... --cluster-replicas 1
     

     Adjust the number of replicas (--cluster-replicas) and list all master/replica nodes you want to include.

By methodically working through these steps, you should be able to diagnose and resolve the "Redis connection refused cluster node down" issue on your Ubuntu 20.04 LTS systems, restoring stability and performance to your applications.