Troubleshooting ‘Redis Connection Refused: Cluster Node Down’ on Alpine Linux

Resolve critical Redis 'connection refused' and cluster node down issues on Alpine Linux. This expert guide covers deep diagnostics and step-by-step fixes.

As an experienced Systems Administrator and DevOps engineer, encountering a "Redis connection refused" error, especially within a cluster on Alpine Linux, signals a critical issue disrupting application performance and data availability. This guide provides a highly technical, step-by-step approach to diagnose and resolve these complex cluster failures, focusing on Alpine's unique characteristics.

Symptom & Error Signature

Users typically experience application errors indicating a failure to connect to Redis, leading to degraded performance or complete service outages. You might see:

1. Application Logs (PHP, Node.js, Python, Ruby etc.):

RedisException: Connection refused in /var/www/html/app/cache_manager.php:123
PHPRedis Exception: Connection refused [tcp://192.168.1.10:6379]

Error: connect ECONNREFUSED 192.168.1.10:6379
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '192.168.1.10',
  port: 6379
}

2. Redis CLI Output:

Attempting to connect via redis-cli might directly show the refusal:

redis-cli -h 192.168.1.10 -p 6379
Could not connect to Redis at 192.168.1.10:6379: Connection refused

When checking cluster status, one or more nodes may be reported as FAIL, PFAIL, or disconnected:

redis-cli --cluster check 192.168.1.10:6379
[ERR] Node 192.168.1.10:6379 is not reachable. Check your firewall rules or that the node is running.

redis-cli -c -p 6379 cluster nodes
# Example showing a failed node
a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0 192.168.1.11:6379@16379 master - 0 1678886400000 1 connected 0-5460
b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0u1 192.168.1.12:6379@16379 master,fail 192.168.1.10:6379 1678886405000 1678886404000 2 disconnected 5461-10922
c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0u1v2 192.168.1.13:6379@16379 master - 0 1678886400000 3 connected 10923-16383

3. System Service Status (Systemd or OpenRC):

If Redis is managed by systemd (common in containerized Alpine or setups with systemd overlay):

systemctl status redis
× redis.service - Redis In-Memory Data Store
     Loaded: loaded (/lib/systemd/system/redis.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Mon 2026-03-15 10:00:00 UTC; 5min ago
       Docs: https://redis.io/docs/
    Process: 1234 ExecStart=/usr/bin/redis-server /etc/redis/redis.conf --daemonize no --supervised systemd (code=exited, status=1/FAILURE)
   Main PID: 1234 (code=exited, status=1/FAILURE)
      CPU: 50ms

For native Alpine installations using OpenRC:

rc-service redis status
 * status: crashed

Root Cause Analysis

The "connection refused" error typically indicates that no process is listening on the specified IP address and port, or a firewall is blocking the connection. In a Redis cluster context, this can be compounded by node failures or misconfigurations.

1. Redis Service Not Running: The most straightforward cause. The redis-server process might have crashed, failed to start, or not been initiated. This could be due to:

   • Configuration Errors: Invalid directives in redis.conf, especially related to bind, port, cluster-enabled, or persistence paths.
   • Resource Exhaustion: Out-Of-Memory (OOM) errors, full disk (for AOF/RDB persistence), or excessive CPU/network load causing the service to become unresponsive or crash.
   • Permissions Issues: Redis cannot read its configuration file, access its log file, or write to its persistence directory (dir).
   • Port Conflict: Another service is already bound to Redis's port (default 6379).

2. Network Connectivity & Firewall:

   • Local Firewall: iptables, ufw (if installed), or cloud provider security groups blocking ingress to Redis's port on the problematic node.
   • Incorrect bind Directive: Redis is configured to listen only on 127.0.0.1 (localhost) while clients are trying to connect from external IPs, or it's bound to an IP that no longer exists on the node.
   • Network Partition/Routing Issues: Between client and server, or between cluster nodes.

3. Redis Cluster State Inconsistency:

   • nodes.conf Corruption: The nodes.conf file, which stores the cluster's topology, might contain incorrect IP addresses (e.g., from a previous network change), be corrupted, or be missing.
   • Split-Brain: Network issues or simultaneous node failures can lead to a cluster partition where different parts of the cluster believe they are the authoritative primary for certain slots, causing instability.
   • Gossip Protocol Failure: Nodes failing to communicate their state to other nodes due to network issues or high load, leading to other nodes marking them as PFAIL or FAIL.

4. Alpine Linux Specifics:

   • Musl libc: While generally compatible, some low-level network libraries or specific Redis builds might behave differently than on glibc-based systems (rare but possible).
   • Default Paths: Alpine's package manager (apk) might install Redis to slightly different paths (/etc/redis/redis.conf, /var/lib/redis) compared to other distributions.
   • Limited System Resources in Containers: If running in a Docker container on Alpine, resource limits (CPU, memory) can be a factor, and the underlying Docker network configuration is crucial.
   • sysctl Configuration: Default sysctl values on Alpine might be more conservative; e.g., net.core.somaxconn (backlog for listen()) might need adjustment for very high connection rates.

Step-by-Step Resolution

Follow these steps meticulously to diagnose and resolve the "Redis connection refused cluster node down" issue on Alpine Linux.

1. Verify Redis Service Status

The first step is to confirm if the Redis server process is actually running on the affected node.

# For Systemd-managed Redis (common in Docker or custom Alpine setups)
systemctl status redis-server.service
# If the service is not 'active (running)', attempt to start it:
systemctl start redis-server.service
# And check its status again. If it fails, proceed to logs.

# For OpenRC-managed Redis (native Alpine default)
rc-service redis status
# If the service is 'stopped' or 'crashed', attempt to start it:
rc-service redis start
# And check its status again. If it fails, proceed to logs.

If the service fails to start or is not running, the output of systemctl status or rc-service status (and journalctl -u redis-server.service for Systemd) will often provide initial clues.

2. Check Network Connectivity & Firewall

Ensure that the problematic node is reachable and its Redis port is open.

# From the client machine trying to connect to Redis node (e.g., 192.168.1.10)
ping 192.168.1.10
# If ping fails, resolve network routing issues first.

# Check if Redis port (default 6379) is open and listening on the Redis node
sudo ss -tulnp | grep 6379
# Expected output (example):
# tcp    LISTEN  0        511          0.0.0.0:6379       0.0.0.0:*      users:(("redis-server",pid=12345,fd=6))
# tcp    LISTEN  0        511              [::]:6379          [::]:*      users:(("redis-server",pid=12345,fd=7))

# If Redis is not listening on the expected IP (e.g., 0.0.0.0 or a specific private IP),
# or if it's only listening on 127.0.0.1, it's a configuration issue (see Step 3).

# Test connectivity from client to server using netcat or telnet
# Replace 192.168.1.10 with your Redis node's IP
nc -vz 192.168.1.10 6379
# Expected successful output:
# Connection to 192.168.1.10 6379 port [tcp/*] succeeded!

# If the connection is refused or timed out, check firewall rules on the Redis node.
# For iptables (common on Alpine):
sudo iptables -L -n | grep 6379
# Look for a rule allowing TCP traffic on port 6379.
# If no explicit allow rule, the default policy might be DROP or REJECT.
# Example command to allow port 6379 (make this persistent with 'apk add iptables-persistent' or similar):
sudo iptables -A INPUT -p tcp --dport 6379 -j ACCEPT
sudo iptables-save > /etc/iptables/rules.v4 # Save rules persistently on Alpine

> [!IMPORTANT]
> Modifying `iptables` directly can lock you out of your server. Always ensure you understand the rules and have console access if applying changes remotely.

#### 3. Review Redis Configuration (`redis.conf`)

Misconfigurations in `redis.conf` are a frequent cause of startup failures or connection issues. The default location on Alpine is typically `/etc/redis/redis.conf`.

```bash
# Open the Redis configuration file
sudo vim /etc/redis/redis.conf

Focus on these critical directives:

• bind:
  • If clients connect from other hosts, bind 127.0.0.1 is problematic.
  • Set bind 0.0.0.0 to listen on all available network interfaces.
  • Alternatively, bind to specific private IPs (e.g., bind 192.168.1.10).
  • Ensure the IP specified is correct for the node.
• protected-mode:
  • For external access, protected-mode no is often necessary if bind is not set to a specific public interface or requirepass is not used.

  • Disabling protected-mode without setting a strong requirepass or firewalling access can expose your Redis instance to the internet, making it vulnerable to attacks.

• port: Ensure this matches the port your applications are trying to connect to.
• cluster-enabled yes: Must be enabled for cluster nodes.
• cluster-config-file nodes.conf: This file stores the cluster's state. Ensure it's pointing to a valid, writable location (default /var/lib/redis/nodes.conf).
• dir /var/lib/redis: The working directory where persistence files (RDB, AOF) and nodes.conf are stored. Ensure this directory exists and Redis has write permissions.

After modifying redis.conf, restart the Redis service:

# For Systemd
sudo systemctl restart redis-server.service

# For OpenRC
sudo rc-service redis restart

Then re-check the service status and attempt to connect with redis-cli.

4. Examine Redis Logs

Redis logs provide crucial insights into why the service might have failed to start or why a node disconnected from the cluster.

# On Systemd-managed systems:
journalctl -u redis-server.service -f

# On OpenRC-managed systems or for direct log file access:
sudo tail -f /var/log/redis/redis-server.log
# (The log file path might vary; check your redis.conf for the 'logfile' directive)

Look for:

• Error messages during startup (e.g., "Error opening config file", "Address already in use").
• OOM (Out-Of-Memory) killer events: Redis might have been killed by the OS due to low memory.
• Disk full errors: If persistence (AOF or RDB) is enabled and the disk reaches capacity.
• Cluster messages: "Adding node", "Removing node", "Handshake failed", "Wrong cluster configuration".
• Permissions errors: "Permission denied" when trying to access nodes.conf or dir.

If you suspect OOM issues, check dmesg -T for "Out of memory" messages related to the Redis process. You may need to increase system memory or adjust Redis's maxmemory setting.

5. Inspect Redis Cluster State & nodes.conf

If the Redis service is running, but cluster connectivity issues persist, the cluster's internal state or the nodes.conf file might be compromised.

# Connect to a *working* node in the cluster (if available) or the problematic node locally
redis-cli -c -h 192.168.1.10 -p 6379 cluster nodes

Analyze the output:

• Look for nodes marked fail, pfail, or disconnected.
• Verify the IP addresses listed for each node. If a node's IP has changed (e.g., due to DHCP or cloud instance redeployment), nodes.conf might contain stale information.
• Check the current_epoch and config_epoch to identify potential split-brain scenarios.

If the IP addresses in nodes.conf are incorrect for a node, you have a few options:

Option A: Forcing Node IP (if service isn't starting due to bad nodes.conf)

If Redis cannot start because nodes.conf references an old IP:

1. Stop Redis on the problematic node.

   sudo systemctl stop redis-server.service # or rc-service redis stop
   

2. Backup the nodes.conf file.

   sudo cp /var/lib/redis/nodes.conf /var/lib/redis/nodes.conf.bak
   

3. Edit nodes.conf manually (extremely cautiously!):
   • Locate the line corresponding to the problematic node.
   • Change its IP address to the current, correct IP.
   • Make sure to also update the bus port (e.g., 192.168.1.10:6379@16379).
   • Alternatively, remove the entire line for the problematic node if you intend to re-add it as a fresh node.
4. Start Redis on the problematic node.

   sudo systemctl start redis-server.service # or rc-service redis start
   

5. Reconnect the node to the cluster (if removed or started with an empty nodes.conf):

   redis-cli -c -h <current_node_ip> -p 6379 cluster meet <known_good_node_ip> <known_good_node_port>
   

Directly editing nodes.conf is highly risky and can lead to data loss or further cluster instability if done incorrectly. Use this only if you understand the file format and the implications. Always back up first.

Option B: Fixing the Cluster using redis-cli --cluster

Redis CLI provides tools to help fix cluster issues:

# Try fixing the cluster from a known good node or the problematic one (if it's accessible)
redis-cli --cluster fix 192.168.1.10:6379

This command attempts to fix a cluster configuration, such as assigning unassigned slots. It may require manual confirmation.

6. Address Resource Exhaustion

Redis is memory-intensive. Resource issues are common, especially on Alpine systems which are often deployed with minimal resources.

• Memory:
  • Check available memory: free -h
  • Monitor Redis memory usage: redis-cli -h 192.168.1.10 -p 6379 info memory
  • Consider increasing system RAM or setting maxmemory in redis.conf to prevent Redis from consuming all available memory.
  • Ensure vm.overcommit_memory = 1 in /etc/sysctl.conf and apply with sysctl -p. This prevents fork failures during RDB saving or AOF rewrites when Redis temporarily doubles its memory usage.
• Disk Space:
  • Check disk usage: df -h
  • Ensure the dir (working directory) in redis.conf has ample free space, especially if AOF or RDB persistence is enabled.

7. Container-Specific Checks (Docker on Alpine)

If Redis is running in a Docker container on Alpine, additional troubleshooting steps are needed:

• Docker Logs: Check the container logs for errors.

  docker logs <container_name_or_id>
  

• Port Mapping: Ensure the Docker container's port 6379 is correctly mapped to the host's port.

  docker ps -a
  # Look for '0.0.0.0:6379->6379/tcp' or similar in the PORTS column.
  

• Docker Network: Verify containers are on the same Docker network if they need to communicate.

  docker inspect <container_name_or_id> | grep Network
  

• Persistent Storage: If nodes.conf or persistence data is lost on container restarts, ensure a Docker volume is mounted for the /data or /var/lib/redis directory within the container.

  # Example Docker Compose snippet
  services:
    redis-node1:
      image: redis:7-alpine
      command: redis-server /usr/local/etc/redis/redis.conf --appendonly yes --cluster-enabled yes
      ports:
        - "6379:6379"
        - "16379:16679" # Cluster bus port
      volumes:
        - ./redis-conf/redis.conf:/usr/local/etc/redis/redis.conf
        - redis-data-node1:/data
  volumes:
    redis-data-node1:
  

  Ensure the nodes.conf file is within the persistent /data directory (or wherever dir points) and not lost on container recreation.

By systematically working through these steps, you should be able to identify and resolve the root cause of "Redis connection refused cluster node down" errors on your Alpine Linux environment. Remember to always apply changes cautiously and validate their impact.