Troubleshooting Redis ‘Connection Refused Cluster Node Down Failure’ on CentOS Stream / Rocky Linux

Resolve Redis connection refused and cluster node failures on CentOS Stream/Rocky Linux. Diagnose binding, firewall, and cluster config issues.

When your applications or redis-cli clients start reporting "connection refused" errors to your Redis cluster nodes, often accompanied by messages like "cluster node down" or general service unavailability, it indicates a critical infrastructure problem. This guide provides a comprehensive, step-by-step approach to diagnose and resolve such issues on CentOS Stream and Rocky Linux environments, focusing on network, configuration, and cluster-specific challenges.

Symptom & Error Signature

Users typically experience application downtime, slow performance, or data access failures. On the terminal or in application logs, you might see error messages similar to these:

Client-side (e.g., redis-cli, application logs):

# redis-cli -h 192.168.1.10 -p 6379
Could not connect to Redis at 192.168.1.10:6379: Connection refused
Error: connect EHOSTUNREACH 192.168.1.10:6379

# Python Redis client example
redis.exceptions.ConnectionRefusedError: Error 111 connecting to 192.168.1.10:6379. Connection refused.

When attempting to query the cluster state, you might get:

# redis-cli -c -h 192.168.1.10 -p 6379 cluster info
(error) CLUSTERDOWN The cluster is down

Redis Server Logs (/var/log/redis/redis.log or journalctl -u redis):

12345:M 13 Aug 2026 10:00:01.123 # Bad file descriptor. Redis is running out of file descriptors.
12345:M 13 Aug 2026 10:00:01.456 # Failed to bind the socket to 127.0.0.1:6379: Address already in use
12345:M 13 Aug 2026 10:00:02.789 # Cluster node config file nodes-6379.conf might be corrupted or not accessible.

Root Cause Analysis

A "connection refused" error typically means that a client attempted to connect to a service on a specific port, but no service was listening on that port, or a firewall actively rejected the connection. In a Redis cluster context, this can be exacerbated by inter-node communication failures. Common root causes include:

1. Redis Service Not Running: The redis-server process might have crashed, failed to start, or been stopped.
2. Firewall Blocking Ports: firewalld (default on CentOS Stream/Rocky Linux) or other network ACLs are preventing client connections to the Redis port (default 6379) or inter-node cluster bus port (default 16379).
3. Incorrect bind Address: Redis is configured to listen only on 127.0.0.1 (localhost), but clients are attempting to connect from other hosts or external interfaces.
4. protected-mode Enabled: Redis's protected-mode (default yes) prevents connections from non-loopback addresses if no bind directive is explicitly set to public interfaces and no requirepass (password) is configured.
5. Redis Cluster Configuration Errors:
   • cluster-enabled no instead of yes.
   • Incorrect cluster-announce-ip or cluster-announce-port directives, especially in multi-NIC or NAT environments.
   • Corrupted or inaccessible nodes-<port>.conf file.
   • Network partition affecting cluster bus communication.
6. Cluster Quorum Failure: If a majority of Redis master nodes are down or unreachable, the cluster may enter a CLUSTERDOWN state, refusing writes.
7. System Resource Limits: Redis might fail to open new connections or perform operations due to reaching limits for open file descriptors (nofile) or memory exhaustion.

Step-by-Step Resolution

Follow these steps on the affected Redis node(s) to diagnose and resolve the connection and cluster issues.

1. Verify Redis Service Status

The first step is to ensure that the Redis service is actually running on the affected node(s).

sudo systemctl status redis

Expected Output (running):

● redis.service - Redis persistent key-value database
     Loaded: loaded (/usr/lib/systemd/system/redis.service; enabled; vendor preset: disabled)
     Active: active (running) since Thu 2026-08-13 09:30:00 UTC; 1min 2s ago
   Main PID: 12345 (redis-server)
     Status: "Ready to accept connections"
      Tasks: 4 (limit: 11096)
     Memory: 10.5M
        CPU: 48ms
     CGroup: /system.slice/redis.service
             └─12345 /usr/bin/redis-server 127.0.0.1:6379 *cluster enabled*

If the service is not running or shows errors, check its recent logs:

sudo journalctl -u redis --since "1 hour ago" -e

If the service is down, attempt to start it:

sudo systemctl start redis
sudo systemctl status redis

If it fails to start, the journalctl output is crucial for identifying the underlying problem.

2. Check Network Connectivity and Firewall

A "connection refused" often points to network or firewall issues.

a. Basic Connectivity Test: From the client machine, try to ping the Redis server IP. If ping fails, you have a fundamental network problem (routing, network interface down, etc.).

ping <redis_node_ip>

b. Port Reachability Test: Use telnet or nc (netcat) to check if the Redis port (6379) and the cluster bus port (16379) are open and listening from the client machine.

# Test Redis data port
telnet <redis_node_ip> 6379

# Test Redis cluster bus port (replace 16379 with your actual cluster bus port if different)
telnet <redis_node_ip> 16379

If telnet immediately says "Connection refused" or hangs, it's likely a firewall or the service isn't listening.

c. Firewall Configuration (firewalld): CentOS Stream and Rocky Linux use firewalld. Check its status and rules on the Redis node.

sudo systemctl status firewalld
sudo firewall-cmd --list-all --zone=public

Look for ports 6379/tcp and 16379/tcp (or your custom Redis ports) listed in the ports section. If they are missing, add them permanently:

> [!IMPORTANT]
> Replace `public` with your active zone if different (e.g., `internal`, `external`). You can check your active zones with `sudo firewall-cmd --get-active-zones`.

sudo firewall-cmd --permanent --zone=public --add-port=6379/tcp
sudo firewall-cmd --permanent --zone=public --add-port=16379/tcp
sudo firewall-cmd --reload

After reloading, re-test with telnet from the client.

3. Inspect Redis Configuration (redis.conf)

The redis.conf file is paramount. Its location is typically /etc/redis/redis.conf or /etc/redis.conf.

sudo vim /etc/redis/redis.conf # Or your specific path

a. bind Directive: By default, Redis might bind only to 127.0.0.1. For external connections, it needs to bind to the specific network interface IP address or 0.0.0.0 (all interfaces).

# Change this for external access
# bind 127.0.0.1 ::1
bind 0.0.0.0 # Binds to all available network interfaces
# OR bind <your_node_private_ip> # Binds to a specific IP address

Binding to 0.0.0.0 without proper firewall rules and Redis authentication (requirepass) is a significant security risk, exposing your Redis instance to the internet. Always implement strong security measures.

b. protected-mode: If bind is set to 127.0.0.1 and protected-mode is yes, Redis will refuse connections from outside localhost.

# If you changed bind to 0.0.0.0 or a specific public IP, you might consider this.
# For security, prefer setting requirepass instead of disabling protected-mode.
# protected-mode yes
protected-mode no

c. cluster-enabled: For a Redis cluster, this must be set to yes.

cluster-enabled yes

d. cluster-announce-ip and cluster-announce-port: These are critical in environments with NAT, Docker, or multiple network interfaces. Redis advertises this IP/port to other cluster nodes. If this is wrong, other nodes will try to connect to the wrong address.

# Uncomment and set your node's externally accessible IP
# cluster-announce-ip 10.0.0.1
cluster-announce-ip <your_node_private_ip> # Or public IP if needed
# cluster-announce-port 6379 # Uncomment if Redis is not listening on default port
# cluster-announce-bus-port 16379 # Uncomment if cluster bus port is not default

After any changes to redis.conf, restart the Redis service:

sudo systemctl restart redis

4. Examine Redis Log Files

The Redis server logs provide detailed information about startup failures, cluster state changes, and errors.

sudo cat /var/log/redis/redis.log | tail -n 100 # Or your specific log path

Look for:

• Lines starting with # for configuration loading issues.
• bind errors ("Address already in use", "Permission denied").
• CLUSTERDOWN messages with specific reasons.
• Messages about nodes.conf corruption or access.

5. Verify Cluster State and nodes.conf

Connect to a working cluster node (if any) or the problematic node (if it's started) using redis-cli in cluster mode to inspect the cluster's health.

redis-cli -c -h <redis_host_ip> -p 6379 cluster info
redis-cli -c -h <redis_host_ip> -p 6379 cluster nodes

Key things to look for in cluster nodes output:

• Nodes marked with fail or fail? indicating they are perceived as down.
• Nodes showing an incorrect IP address (compare with cluster-announce-ip).
• The nodes-<port>.conf file contains the authoritative cluster configuration for each node. If this file becomes corrupted or has incorrect entries, a node might refuse to join or operate correctly. It's usually located in the Redis data directory (e.g., /var/lib/redis/nodes-6379.conf).

Directly modifying or deleting nodes-<port>.conf should be a last resort and only done with extreme caution. Incorrect handling can lead to permanent data loss or cluster instability. Always back up the file first.

If a node's nodes-<port>.conf is suspected to be corrupted, or if you are trying to reintegrate a node that has been offline for a long time and its cluster view is outdated:

1. Stop Redis on the problematic node:

   sudo systemctl stop redis
   

2. Backup the nodes-<port>.conf file:

   sudo cp /var/lib/redis/nodes-6379.conf /var/lib/redis/nodes-6379.conf.backup_$(date +%F-%H%M)
   

3. Delete the nodes-<port>.conf file: This will make Redis start as a fresh node, attempting to join an existing cluster.

   sudo rm /var/lib/redis/nodes-6379.conf
   

4. Start Redis:

   sudo systemctl start redis
   

5. Re-add the node to the cluster (if it's a new or previously removed node):

   redis-cli --cluster add-node <new_node_ip>:6379 <existing_node_ip>:6379
   

   This command connects the new node to an existing node, which then helps it discover and join the cluster.

6. Handle Cluster Quorum Issues

If a majority of your master nodes are down, the entire cluster might enter a CLUSTERDOWN state and stop accepting writes. This is a safety mechanism.

• Bring more nodes online: The primary solution is to bring enough master nodes back online to achieve quorum.

• Force a new configuration (last resort): If you've lost too many masters and cannot recover them, you might be able to force the remaining masters to elect a new configuration. This can lead to data loss if not all masters are synchronized.

  > [!WARNING]
  > Use `redis-cli --cluster fix` with extreme caution. It can lead to data loss if not properly understood and executed. Only use this if you are absolutely sure about the state of your data and understand the implications.
  
  redis-cli --cluster fix <ip_of_a_surviving_master_node>:6379
  

7. Increase System Resource Limits

Redis needs sufficient file descriptors for connections, especially in a cluster with many clients or nodes.

a. Check current limits: On the Redis node, check the nofile limit for the Redis process.

cat /proc/<redis_pid>/limits | grep "Max open files" # Replace <redis_pid> with actual PID from `systemctl status redis`

b. Increase system-wide limits: Edit /etc/sysctl.conf to increase the maximum number of file handles.

sudo vim /etc/sysctl.conf

Add or modify:

fs.file-max = 200000

Apply the change:

sudo sysctl -p

c. Increase user/service limits: Edit /etc/security/limits.conf to set nofile limits for the redis user.

sudo vim /etc/security/limits.conf

Add or modify:

redis soft nofile 65536
redis hard nofile 65536

For limits.conf changes to take effect, the redis service often needs to be restarted. In some cases, a full reboot might be necessary or ensure that UsePAM yes is set in /etc/ssh/sshd_config and session required pam_limits.so is in /etc/pam.d/system-auth and /etc/pam.d/sshd.

8. Redeploy/Rebuild a Failed Cluster Node

If a node is consistently failing and unrecoverable through the above steps, you might need to treat it as a new node.

1. Stop Redis on the problematic node.
2. Backup Data & Configuration:

   sudo systemctl stop redis
   sudo cp -r /var/lib/redis /var/lib/redis.backup_$(date +%F-%H%M)
   sudo cp /etc/redis/redis.conf /etc/redis/redis.conf.backup_$(date +%F-%H%M)
   

3. Clean Redis data and config files (only if you intend to provision it as a truly new node, losing any unique data it might have contained if not recovered from another replica).

   sudo rm -rf /var/lib/redis/*
   sudo rm -f /var/log/redis/redis.log # Optional, for fresh logs
   

4. Ensure redis.conf is correctly configured (especially cluster-enabled yes and cluster-announce-ip).
5. Start Redis:

   sudo systemctl start redis
   

6. Add the node back to the cluster: If it's a new master:

   redis-cli --cluster add-node <new_node_ip>:6379 <existing_node_ip>:6379
   redis-cli --cluster reshard <existing_node_ip>:6379 # To assign slots
   

   If it's a new replica for an existing master:

   redis-cli --cluster add-node <new_replica_ip>:6379 <existing_master_ip>:6379 --cluster-slave --cluster-master-id <master_node_id>
   

   You can get <master_node_id> from redis-cli -c -h <existing_master_ip> -p 6379 cluster nodes.

By systematically working through these troubleshooting steps, you can effectively diagnose and resolve "Redis connection refused cluster node down" issues on your CentOS Stream or Rocky Linux hosting environment.