Troubleshooting ‘Git SSH connection closed by foreign host port 22 fatal protocol’ on Alpine Linux

Resolve 'Git SSH connection closed by foreign host port 22' with 'fatal protocol' errors on Alpine Linux. This guide covers firewall rules, SSH daemon issues, and key authentication for seamless Git operations.

Introduction

As an experienced Systems Administrator and DevOps engineer, you know that seamless Git operations are crucial for development and deployment workflows. Encountering the error message "'Git SSH connection closed by foreign host port 22 fatal protocol' on Alpine Linux" can be a significant roadblock. This issue typically indicates an abrupt termination of the SSH connection by the remote server, often occurring during the initial handshake or before successful authentication, making it impossible to perform Git actions like cloning, pulling, or pushing.

This guide provides a highly technical, accurate, and step-by-step approach to diagnose and resolve this complex problem, focusing on Alpine Linux-specific configurations and general SSH best practices.

Symptom & Error Signature

When attempting to interact with a Git repository over SSH (e.g., git clone, git fetch, git push), you will typically see output similar to one of the following variations in your terminal:

$ git clone git@your-alpine-server:path/to/repo.git
Cloning into 'repo'...
ssh_exchange_identification: read: Connection closed by remote host
fatal: protocol error: bad passkey
fatal: The remote end hung up unexpectedly

Another common manifestation, particularly when debugging with verbose SSH output, might look like:

$ ssh -vvv git@your-alpine-server
OpenSSH_8.x, OpenSSL 1.x.y  zz-Mon-YYYY
debug1: Reading configuration data /home/user/.ssh/config
debug1: /home/user/.ssh/config line X: Applying options for your-alpine-server
debug1: Authenticator provider "internal"
debug1: Connecting to your-alpine-server [X.X.X.X] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/id_rsa type 0
debug1: identity file /home/user/.ssh/id_dsa type -1
debug1: identity file /home/user/.ssh/id_ecdsa type -1
debug1: identity file /home/user/.ssh/id_ed25519 type -1
debug1: identity file /home/user/.ssh/id_xmss type -1
debug1: Local version string SSH-2.0-OpenSSH_8.x
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.x
debug1: compat_banner: no match: OpenSSH_8.x
debug1: Skipping SSH2_MSG_KEXINIT - old key exchange, rekeying is not supported
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/user/.ssh/id_rsa RSA SHA256:XXXXXXXXX agent
debug3: send packet: type 50
debug1: send_pubkey_test: no mutual signature algorithm
debug3: send packet: type 50
debug2: we did not send a packet, disable key
debug1: send_pubkey_test: no mutual signature algorithm
debug3: send packet: type 50
debug2: we did not send a packet, disable key
debug1: Connection closed by your-alpine-server port 22
ssh_exchange_identification: read: Connection closed by remote host

The key indicators are "Connection closed by remote host" and "fatal: protocol error," which collectively point to an issue where the remote SSH server terminated the connection prematurely.

Root Cause Analysis

This error can stem from various issues, primarily on the server side, but occasionally from client misconfigurations or network intermediaries. The "fatal protocol" aspect often implies a failure during the initial SSH negotiation or an immediate server rejection based on policy.

1. Firewall Restrictions: The most common culprit. A firewall (e.g., iptables, nftables on Alpine, cloud security groups, or network devices) on the client, server, or in between, is blocking or dropping packets to/from port 22, preventing the SSH connection from fully establishing or immediately terminating it.
2. Server-Side SSH Daemon (sshd) Issues:
   • sshd service is not running or has crashed on the Alpine host.
   • Incorrect sshd_config settings: This includes ListenAddress, Port, Protocol, AllowUsers, DenyUsers, MaxStartups, or specific, overly restrictive Ciphers, KexAlgorithms, or HostKeyAlgorithms that lead to negotiation failure.
   • sshd reaching MaxStartups limits for unauthenticated connections, causing new connections to be dropped.
   • sshd process exiting prematurely due to resource constraints (low memory, CPU, file descriptors) or a corrupt installation.
3. SSH Protocol Negotiation Failure: The "fatal protocol" message strongly suggests that the SSH client and server failed to agree on a common set of algorithms for key exchange, host key signature, or encryption. This can be due to outdated SSH client/server versions or a highly restrictive sshd_config on the server not supporting modern/client-offered algorithms.
4. SSH Key Authentication Problems (Edge Cases): While typically resulting in "Permission denied," severe issues like non-existent keys, incorrect permissions on ~/.ssh/authorized_keys, or server-side restrictions on specific key types can sometimes lead to an immediate disconnect, particularly if the server terminates the connection early rather than sending a detailed authentication failure message.
5. fail2ban or IP Blocking Software: The client's IP address might have been temporarily or permanently banned by fail2ban or similar intrusion prevention systems on the Alpine server due to excessive failed login attempts.
6. Network Intermediaries/Proxies: Firewalls, load balancers, or proxy servers might be inspecting or tampering with SSH traffic, leading to a malformed protocol negotiation or premature connection termination.

Step-by-Step Resolution

Follow these steps meticulously, performing checks on both your client machine and the Alpine Linux server.

1. Initial Network & Basic Connectivity Checks

Always start with the basics to ensure network reachability.

• Ping the Server: From your client machine, ensure the Alpine server is reachable.

  ping your-alpine-server.example.com
  

  If ping fails, check DNS resolution, network cables, and basic network configuration.
• Check SSH Port Reachability: Use telnet or netcat to verify that port 22 (or your custom SSH port) on the server is open and responding.

  telnet your-alpine-server.example.com 22
  # OR (if telnet is not installed or preferred)
  nc -vz your-alpine-server.example.com 22
  

  • If you see Connection refused or No route to host, it indicates either a firewall blocking the connection or the sshd service not running on the server.
  • If it connects (Connected to ...) and then immediately closes, it strongly suggests the server's SSH daemon is intentionally or unintentionally dropping the connection early, or an aggressive firewall is in play.

2. Server-Side SSH Daemon (sshd) Status on Alpine

You will need console access or another working SSH session to the Alpine server for these steps.

• Check sshd Service Status: Verify that the OpenSSH server daemon is running.

  rc-service sshd status
  

  If it's not running, start it and enable it to start on boot:

  rc-service sshd start
  rc-update add sshd default
  

• Inspect SSH Logs: Check the system logs for any errors related to sshd, connection attempts, or disconnections.

  grep -i ssh /var/log/messages # Common Alpine log location
  # Depending on your syslog configuration, also check:
  # grep -i ssh /var/log/auth.log
  # grep -i ssh /var/log/secure
  

  Look for messages indicating why connections are being dropped or if sshd is failing to start.

3. Inspect Server-Side SSH Configuration (sshd_config)

Misconfigurations in /etc/ssh/sshd_config are a frequent cause of "connection closed" errors.

• Backup sshd_config: Always create a backup before making changes.

  cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak
  

• Review Essential Settings in /etc/ssh/sshd_config: Open the file with your preferred editor (vi or nano).

  vi /etc/ssh/sshd_config
  

  Pay close attention to the following directives:

  • Port 22: Ensure it matches the port you're trying to connect to.
  • ListenAddress 0.0.0.0 or ListenAddress <Server_IP>: Confirm sshd is listening on the correct network interface.
  • Protocol 2: Ensure only SSHv2 is enabled (recommended).
  • PubkeyAuthentication yes: Crucial for key-based Git authentication.
  • PasswordAuthentication no (if using keys only, otherwise yes): Ensure this matches your authentication method.
  • PermitRootLogin no: Standard security practice.
  • AllowUsers, DenyUsers, AllowGroups, DenyGroups: Verify your Git user is not explicitly denied.
  • MaxStartups: This is a critical setting that limits the number of concurrent unauthenticated connections. If too low (e.g., 1), or if many concurrent connections are being attempted, sshd will drop new connections. The default is usually 10:30:100 (10 unauthenticated connections, then 30% chance of dropping, up to 100). If this line is uncommented and set to a very low value, try increasing it or commenting it out to use the default.
  • Algorithm Restrictions: Look for Ciphers, KexAlgorithms, or HostKeyAlgorithms directives. If these are present and overly restrictive (e.g., listing only very old or very new algorithms), your client might not be able to find a common negotiation point. Try commenting them out temporarily to use sshd's defaults.
  • UsePAM yes / AuthenticationMethods: If PAM is heavily configured or AuthenticationMethods explicitly excludes publickey, it could cause issues.

  After any changes to /etc/ssh/sshd_config, you must reload or restart the SSH daemon for changes to take effect.

  rc-service sshd reload # Preferred for non-disruptive changes
  # If reload fails or for more significant changes (e.g., port change), use:
  # rc-service sshd restart
  

  Always test from a separate terminal session or console after restarting sshd to ensure you don't lock yourself out of the server.

4. Check Server-Side Firewall Rules (Alpine iptables/nftables)

Alpine Linux commonly uses iptables or nftables directly for firewall management.

• List current iptables rules:

  # Install iptables if not present
  apk add iptables
  iptables -L -v -n
  

  Look carefully for any DROP or REJECT rules affecting TCP traffic on port 22 in the INPUT chain. Ensure there is an ACCEPT rule for TCP traffic on port 22. A typical correct rule might look like: -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT.

• List current nftables rules (if used):

  # Install nftables if not present
  apk add nftables
  nft list ruleset
  

  Search for similar tcp dport 22 accept rules.

  Incorrectly modifying firewall rules can lock you out of your server. Exercise extreme caution. If you must add a rule, make it as specific as possible (e.g., restrict by source IP if known).

  To temporarily allow SSH from anywhere (for testing purposes only, not recommended for production):

  # DANGEROUS: Opens port 22 to the world!
  iptables -A INPUT -p tcp --dport 22 -j ACCEPT
  # Remember to persist rules if you want them to survive a reboot.
  # For iptables: iptables-save > /etc/iptables/rules.v4 (and ensure iptables service is enabled: rc-update add iptables default)
  

  Cloud provider security groups (e.g., AWS Security Groups, Azure Network Security Groups, Google Cloud Firewall Rules) can also block port 22. Ensure these are configured to allow ingress traffic on port 22 from your client's IP address.

5. Investigate fail2ban or IP Banning Software

If fail2ban is installed on your Alpine server, your client's IP might be temporarily or permanently banned due to too many failed login attempts.

• Check fail2ban status and banned IPs:

  # Install fail2ban if not present
  apk add fail2ban
  fail2ban-client status sshd # Or the specific jail protecting SSH
  fail2ban-client unban <YOUR_CLIENT_IP> # Unban your IP if found
  

  Also, check /var/log/auth.log for messages like "Failed password for" or "Disconnect from" originating from your client IP, which could indicate a ban.

6. Client-Side SSH Debugging & Configuration

While the issue is often server-side, client configurations can sometimes contribute.

• SSH with Maximum Verbosity: This command is your best friend for debugging SSH connections. It provides detailed output on the client's attempt to connect and negotiate.

  ssh -vvv git@your-alpine-server
  

  Carefully examine the output for lines immediately preceding "Connection closed by remote host" or "fatal protocol error." Look for:
  • no mutual signature algorithm: Indicates client/server can't agree on a key exchange or host key algorithm.
  • Messages about authentication methods being rejected early.
  • Any unusual "debug" messages hinting at protocol deviations.
• Check ~/.ssh/config: Temporarily comment out or remove any host-specific configurations for your-alpine-server in your ~/.ssh/config file to rule out client-side misconfigurations.

  # Example: comment out problematic lines
  # Host your-alpine-server
  #   User git
  #   IdentityFile ~/.ssh/problematic_key
  #   Ciphers some_ancient_cipher
  

• SSH Agent and Keys: Ensure your SSH agent is running and has the correct private key loaded.

  eval "$(ssh-agent -s)" # Start agent if not running
  ssh-add -l             # List loaded keys
  ssh-add ~/.ssh/id_rsa  # Add your private key if not listed (replace with actual key path)
  

7. SSH Key Pair Integrity and Permissions

Incorrect permissions on your SSH keys or the .ssh directory can cause sshd to reject connections instantly for security reasons.

• Client-side (~/.ssh/):

  chmod 700 ~/.ssh
  chmod 600 ~/.ssh/id_rsa    # Private key
  chmod 644 ~/.ssh/id_rsa.pub # Public key
  

• Server-side (~/.ssh/ for the Git user): For the user you're connecting as on the Alpine server (e.g., git user for Git operations):

  # Assuming your Git user is named 'git'
  chown git:git /home/git/.ssh
  chmod 700 /home/git/.ssh
  chown git:git /home/git/.ssh/authorized_keys
  chmod 600 /home/git/.ssh/authorized_keys
  

  The permissions on ~/.ssh and ~/.ssh/authorized_keys on the server are critically important. If they are too permissive, sshd will refuse to use them, leading to connection failures.

By methodically working through these steps, you should be able to pinpoint the exact cause of the "Git SSH connection closed by foreign host port 22 fatal protocol" error on your Alpine Linux system and restore seamless Git operations.