Troubleshooting Redis ‘Connection Refused’ & Cluster Node Down on Ubuntu 22.04 LTS

Resolve Redis 'Connection refused' and cluster node down issues on Ubuntu 22.04 LTS. This guide covers common causes like firewall, configuration, and cluster misconfigurations.

When managing high-performance web applications, Redis is often the backbone for caching, session management, and real-time data processing. Encountering a "Redis connection refused" error, especially within a cluster environment, can bring your services to a halt. This comprehensive guide will equip experienced Systems Administrators and DevOps engineers with the knowledge to diagnose and rectify these critical issues on Ubuntu 22.04 LTS.

The primary symptom you'll likely observe is your application failing to connect to Redis, resulting in errors like database connection failures, session loss, or a complete inability to serve content that relies on Redis. In a cluster setup, this might propagate to other nodes or even lead to a cluster partition.

Symptom & Error Signature

Applications attempting to connect to Redis will typically display errors similar to these:

Application Log / PHP Error:

PHPRedisException: Redis connection refused in /var/www/html/app/cache.php:25
Stack trace:
#0 /var/www/html/app/cache.php(25): Redis->connect('127.0.0.1', 6379)
#1 /var/www/html/index.php(10): CacheService->init()
#2 {main}

// Or for a remote connection
Could not connect to Redis at 10.0.0.5:6379: Connection refused

redis-cli Output:

$ redis-cli -h 127.0.0.1 -p 6379
Could not connect to Redis at 127.0.0.1:6379: Connection refused
not connected>

$ redis-cli -c -h 10.0.0.5 -p 6379
(error) ERR Cluster node 10.0.0.5:6379 is down

System Log (journalctl) for Redis Service Failure:

systemd[1]: redis-server.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: redis-server.service: Failed with result 'exit-code'.
systemd[1]: redis-server.service: Consumed 1.056s CPU time.

Root Cause Analysis

A "connection refused" error typically indicates that the client attempted to establish a TCP connection, but the server actively rejected it or no process was listening on the specified port. In a Redis cluster context, this can be more complex. Here's a breakdown of common underlying reasons:

1. Redis Service Not Running: The most straightforward cause. The redis-server process might have crashed, failed to start, or was manually stopped.
2. Network Accessibility & Firewall:
   • Firewall Rules: ufw, iptables, or cloud security groups are blocking incoming connections to Redis's port (default 6379).
   • Incorrect bind Directive: The bind directive in redis.conf might be set to 127.0.0.1 (localhost only), preventing remote connections.
   • protected-mode: If protected-mode yes is enabled and bind is 127.0.0.1, remote connections will be refused unless authenticated.
3. Redis Configuration Errors (redis.conf):
   • Mismatched Port: Client attempts to connect on a port different from what Redis is configured to listen on.
   • Cluster Misconfiguration: In a cluster, nodes might have inconsistent cluster-enabled settings, incorrect cluster-config-file paths, or incorrect node IP addresses.
   • requirepass Mismatch: If Redis authentication is enabled, incorrect or missing password.
4. Resource Exhaustion:
   • Out of Memory (OOM): The operating system's OOM Killer might have terminated the Redis process due to high memory consumption.
   • File Descriptor Limits: Redis might exceed the allowed number of open file descriptors, preventing it from accepting new connections.
5. Corrupted Persistence Files: If appendonly.aof or dump.rdb files are corrupted, Redis might fail to start, leading to a connection refused error.
6. Redis Cluster Node Health:
   • Quorum Loss: Too many master nodes are down, preventing the cluster from operating.
   • Partitioning: Network issues or misconfigurations cause nodes to lose communication, leading to a fragmented cluster.
   • Node IP Changes: If a node's IP address changes (e.g., in a cloud environment), it needs to be updated in the cluster configuration.

Step-by-Step Resolution

Follow these steps meticulously to diagnose and resolve your Redis connection issues.

1. Verify Redis Service Status

The first step is always to ensure the Redis server process is actually running.

sudo systemctl status redis-server

Expected Output for a healthy service:

● redis-server.service - Advanced key-value store
     Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2026-10-26 10:00:00 UTC; 1min 2s ago
       Docs: http://redis.io/documentation, man:redis-server(1)
    Process: 1234 ExecStart=/usr/bin/redis-server /etc/redis/redis.conf --daemonize no --supervised systemd (code=exited, status=0/SUCCESS)
   Main PID: 1235 (redis-server)
      Tasks: 4 (limit: 4627)
     Memory: 10.5M
        CPU: 45ms
     CGroup: /system.slice/redis-server.service
             └─1235 /usr/bin/redis-server 127.0.0.1:6379 *cluster

If the status is inactive (dead), failed, or activating (auto-restart), proceed to restart the service and check logs:

sudo systemctl start redis-server
sudo systemctl status redis-server
sudo journalctl -xeu redis-server | tail -n 50

If systemctl start redis-server fails, the journalctl output is crucial. Look for error messages related to configuration, memory, or file corruption.

2. Check Network Connectivity & Firewall

If Redis is running but clients still report "connection refused," the issue is likely network-related.

A. Verify Redis is Listening: From the Redis server itself, check if Redis is listening on the expected port and IP address.

sudo ss -tuln | grep 6379

Expected Output:

tcp   LISTEN 0      4096   127.0.0.1:6379      0.0.0.0:*
tcp   LISTEN 0      4096   10.0.0.5:6379       0.0.0.0:*  # Example for specific IP
tcp   LISTEN 0      4096   0.0.0.0:6379        0.0.0.0:*  # Example for all interfaces

If you see 127.0.0.1:6379 but your client is remote, that's a key indicator. If no output is shown for port 6379, Redis is not listening, likely due to a service failure or misconfiguration.

B. Test Connectivity from Client: Use telnet or netcat from the client machine to test connectivity to the Redis server's IP and port.

# From the client machine
telnet <REDIS_SERVER_IP> 6379
# Or
nc -vz <REDIS_SERVER_IP> 6379

If telnet immediately shows "Connection refused" or nc reports "Connection refused," a firewall or bind address issue is highly probable. If it hangs, it might be a firewall blocking the connection.

C. Inspect Firewall Rules:

• UFW (Uncomplicated Firewall) on Redis Server:

  sudo ufw status verbose
  

  Ensure port 6379/tcp is allowed from the client's IP or from anywhere (Anywhere).

  # To allow from a specific IP address
  sudo ufw allow from <CLIENT_IP_ADDRESS> to any port 6379
  
  # To allow from any IP (use with caution, only on private networks or with other security measures)
  sudo ufw allow 6379/tcp
  
  sudo ufw reload
  

• Cloud Security Groups/Network ACLs: If running in a cloud environment (AWS EC2, Google Cloud, Azure VM), check the associated security groups or network ACLs. Ensure inbound traffic on port 6379 is permitted from your client's IP range.

3. Review redis.conf Configuration

The primary Redis configuration file on Ubuntu is typically located at /etc/redis/redis.conf.

sudo nano /etc/redis/redis.conf

A. bind Directive: Look for the bind directive.

• If set to bind 127.0.0.1, Redis will only accept connections from the local machine. For remote clients, you must change this.
  • To bind to all available network interfaces (use with caution and ensure firewall is robust):

    bind 0.0.0.0
    

  • To bind to specific IP addresses (e.g., your server's private IP for cluster communication):

    bind 127.0.0.1 10.0.0.5 # Example for local and a specific private IP
    

Binding to 0.0.0.0 without proper firewall rules exposes your Redis instance to the entire internet, which is a significant security risk. Only do this if you are absolutely certain your network configuration provides adequate protection (e.g., within a private VPC with strict security groups).

B. protected-mode: If bind is not 0.0.0.0 or a specific network interface, and you're getting connection refused errors, check protected-mode.

protected-mode yes

If protected-mode is yes and bind is 127.0.0.1 (or commented out), Redis will only accept connections from localhost unless you specify a requirepass. For remote access, either configure bind appropriately or set protected-mode no (less secure without requirepass).

protected-mode no # Use with caution and consider 'requirepass'

C. port Directive: Ensure the port matches what your client applications are trying to connect to. Default is 6379.

port 6379

D. requirepass (Authentication): If requirepass is uncommented and set, your clients must provide the correct password. An incorrect or missing password from the client side can manifest as a "connection refused" if the client library doesn't handle the authentication failure gracefully, though often it's a specific authentication error.

requirepass your_strong_redis_password

If you uncomment or change requirepass, remember to update all client applications and other cluster nodes accordingly.

After any changes to redis.conf, restart Redis:

sudo systemctl restart redis-server

4. Investigate Redis Cluster Specific Issues

If your setup involves a Redis Cluster, more specific checks are needed.

A. Check Cluster Health: Use redis-cli on any node to get the cluster status.

redis-cli -c -h <NODE_IP> -p <NODE_PORT> cluster nodes
redis-cli -c -h <NODE_IP> -p <NODE_PORT> cluster info

Look for nodes marked fail, fail?, or disconnected. The cluster info command will provide a high-level overview, including cluster_state, cluster_known_nodes, and cluster_fail_messages.

B. Node Reachability within Cluster: Ensure all nodes can communicate with each other on the Redis cluster bus port (default 16379, i.e., port + 10000) and the regular client port (6379). Check firewalls on all cluster nodes for both ports.

C. cluster-config-file Integrity: Each cluster node has a nodes.conf (or similar, as defined by cluster-config-file in redis.conf) file that stores its cluster state. If this file is corrupted, Redis might fail to start or join the cluster.

# Find the path from redis.conf or logs
sudo cat /var/lib/redis/nodes-6379.conf

If the file is empty or malformed, it could be an issue. In severe cases, deleting this file (while Redis is stopped) can force a node to rejoin as a new node, but this carries a risk of data loss or cluster re-shuffling.

Deleting nodes.conf should be a last resort and understood as a potentially destructive action. Always have a robust backup strategy.

D. IP Address Changes for Cluster Nodes: If a server's IP address changes, the cluster configuration needs to be updated. You can use redis-cli cluster forget <node_id> on all healthy nodes, then redis-cli cluster meet <new_node_ip> <new_node_port> from the affected node to rejoin.

5. Check for Resource Exhaustion (OOM, File Descriptors)

If Redis is frequently crashing or failing to start, resource limits could be the cause.

A. Out Of Memory (OOM) Killer: Check the kernel message buffer for OOM Killer events:

dmesg -T | grep -i oom

If you see entries like Out of memory: Killed process 1234 (redis-server), your server ran out of RAM. Resolution:

• Increase server RAM.
• Optimize Redis memory usage (e.g., use more efficient data structures, set maxmemory directive in redis.conf).
• Configure maxmemory-policy in redis.conf (e.g., allkeys-lru to evict old keys).

B. File Descriptor Limits: Redis requires a file descriptor for each client connection and internal files. If it exceeds system limits, it can fail.

• Check system-wide limits:

  sysctl fs.file-max
  

• Check Redis process limits (if running):

  cat /proc/<REDIS_PID>/limits | grep "Max open files"
  

  (Replace <REDIS_PID> with the actual PID of the redis-server process from systemctl status redis-server.)
• Adjust limits:
  • For the Redis service, edit the systemd service file:

    sudo systemctl edit redis-server
    

    Add/modify LimitNOFILE:

    [Service]
    LimitNOFILE=65535 # Example value, adjust as needed
    

    Save, then reload systemd and restart Redis:

    sudo systemctl daemon-reload
    sudo systemctl restart redis-server
    

  • Alternatively, edit /etc/security/limits.conf (requires re-login or system reboot for some services).

6. Handle AOF/RDB Persistence File Corruption

If Redis fails to start and logs indicate issues loading appendonly.aof or dump.rdb, these files might be corrupted.

A. Backup Persistence Files: Before attempting any fixes, always back up these files.

sudo cp /var/lib/redis/appendonly.aof /var/lib/redis/appendonly.aof.bak
sudo cp /var/lib/redis/dump.rdb /var/lib/redis/dump.rdb.bak

(Adjust paths based on your dir and dbfilename in redis.conf).

B. Fix AOF File: Redis provides a utility to fix corrupted AOF files:

sudo redis-check-aof --fix /var/lib/redis/appendonly.aof

This tool will attempt to truncate the AOF file at the first point of corruption, potentially leading to data loss but allowing Redis to start.

C. Remove Corrupted Files (Data Loss Risk): As a last resort, if you can tolerate data loss or have recent backups, you can move or delete the persistence files to force Redis to start with an empty dataset.

sudo systemctl stop redis-server
sudo mv /var/lib/redis/appendonly.aof /var/lib/redis/appendonly.aof.corrupted
sudo mv /var/lib/redis/dump.rdb /var/lib/redis/dump.rdb.corrupted
sudo systemctl start redis-server

Removing or fixing persistence files can lead to data loss. Ensure you have a backup strategy in place and understand the implications before proceeding.

By systematically working through these steps, you should be able to identify and resolve the "Redis connection refused cluster node down" issue on your Ubuntu 22.04 LTS servers.