Troubleshooting ‘Git error pack-objects died of signal 13 pipe break’ during Large Pushes on Ubuntu 20.04 LTS

Resolve 'pack-objects died of signal 13' Git errors during large pushes on Ubuntu 20.04 LTS. This guide addresses SSH timeouts, HTTP buffer limits, and Git pack configurations.

Introduction

Encountering a "Git error pack-objects died of signal 13 pipe break" message during a large git push operation can be a frustrating experience for developers and system administrators. This cryptic error indicates that the pack-objects process, responsible for efficiently compressing and transferring Git objects, received a SIGPIPE signal, meaning one end of the communication pipe closed unexpectedly while the other was still attempting to write data. This often occurs when pushing repositories with a substantial number of files, very large individual files, or extensive commit histories, as the transfer duration and resource demands increase, making the connection more susceptible to timeouts or resource exhaustion.

This guide provides a comprehensive, highly technical approach to diagnose and resolve this issue on Ubuntu 20.04 LTS systems, covering common culprits from SSH server configurations to Git client and server settings.

Symptom & Error Signature

When attempting to push a large volume of data to a remote Git repository, typically via SSH or HTTPS, the push operation fails with an error similar to the following:

Enumerating objects: 1234567, done.
Counting objects: 100% (1234567/1234567), done.
Delta compression using up to 8 threads
Compressing objects: 100% (1234560/1234560), done.
Writing objects: 100% (1234567/1234567), 9.87 GiB | 10.00 MiB/s, done.
Total 1234567 (delta 987654), reused 1234567 (delta 987654)
error: remote unpack failed: unable to create temporary file: No space left on device
fatal: the remote end hung up unexpectedly
fatal: the remote end hung up unexpectedly
error: pack-objects died of signal 13

The specific error error: pack-objects died of signal 13 is the key indicator, often accompanied by fatal: the remote end hung up unexpectedly or other remote-related messages. While No space left on device might appear in some contexts, the SIGPIPE (signal 13) error specifically points to a broken communication channel rather than storage capacity.

Root Cause Analysis

The "pack-objects died of signal 13" error is fundamentally a SIGPIPE (Pipe Broken) signal received by the pack-objects process. This signal indicates that the process attempted to write to a pipe or socket whose reading end has been closed. In the context of Git pushes, this can stem from several underlying issues:

1. SSH Server Timeouts (Most Common for SSH Remotes):

   • During prolonged data transfers, the SSH server (sshd) might terminate an "inactive" connection if no activity (or keep-alive packets) is observed for a configured duration. This is governed by ClientAliveInterval and ClientAliveCountMax in sshd_config. When the SSH connection is closed prematurely, the pipe breaks, and the client-side pack-objects process receives SIGPIPE.

2. HTTP/S postBuffer Limits (Most Common for HTTP/S Remotes):

   • When pushing over HTTP/S, Git clients use HTTP POST requests to send packed objects. Web servers (e.g., Nginx, Apache) or Git's internal HTTP server might have limitations on the size of the request body or an upload timeout. If the packed objects exceed the http.postBuffer setting on the client or the server, or if the transfer takes too long, the connection can be reset.

3. Network Instability or Intermediate Device Timeouts:

   • Firewalls, proxies, load balancers, or NAT devices between the client and the Git server often have their own idle connection timeouts. A long-running Git push, especially over a flaky network, can exceed these timeouts, leading to a connection reset and a SIGPIPE.

4. Server-side Resource Exhaustion:

   • While less direct for SIGPIPE, severe memory pressure, disk I/O bottlenecks, or hitting ulimit restrictions on the Git server can cause processes to hang, be killed by the OOM (Out Of Memory) killer, or fail to respond, indirectly leading to the client's connection breaking and a SIGPIPE. The pack-objects process can be memory-intensive.

5. Corrupted Repository or Git Installation:

   • Though rarer for SIGPIPE, a severely corrupted local or remote repository, or issues with the Git installation itself, could theoretically lead to unexpected process termination.

Step-by-Step Resolution

Address these issues methodically, starting with the most common causes based on your remote protocol (SSH or HTTP/S).

1. Adjust Git HTTP Post Buffer (Client-side)

If you are pushing over HTTP/S, the client's http.postBuffer might be too small for large packs. This setting dictates the maximum size of the HTTP POST buffer Git will use when sending data.

1. Increase http.postBuffer globally: Open your global Git configuration file (~/.gitconfig):

   git config --global http.postBuffer 524288000 # 500 MB
   

   You can adjust 524288000 (bytes) to a larger value like 1048576000 (1 GB) if necessary.

2. Increase http.postBuffer for a specific repository: Navigate to your repository and set it locally:

   cd /path/to/your/repo
   git config http.postBuffer 524288000
   

This change only affects HTTP/S pushes. If your remote URL starts with git@ or ssh://, this setting will not apply.

2. Adjust Git HTTP Post Buffer (Server-side)

If your Git server uses HTTP/S (e.g., self-hosted GitLab, Gitea, or a plain Git daemon fronted by Nginx/Apache), the server-side http.postBuffer or web server upload limits might be the bottleneck.

1. For Git servers using Git smart HTTP: On the server, if you manage a raw Git HTTP endpoint, you might need to adjust http.postBuffer in the server's global Git config or repository config, similar to the client-side. Alternatively, check the web server configuration.

2. For Nginx (Common for GitLab/Gitea): Increase the client_max_body_size directive in your Nginx configuration for the Git host. This is typically found in /etc/nginx/nginx.conf, /etc/nginx/sites-available/your-git-host, or similar.

   # In http, server, or location block
   client_max_body_size 500M; # Or 1G, 2G depending on your needs
   client_body_timeout 300s; # Increase timeout for large uploads
   send_timeout 300s;
   

   After modification, test the Nginx configuration and reload/restart:

   sudo nginx -t
   sudo systemctl reload nginx
   

3. For Apache (Less common for modern Git services but applicable): Adjust LimitRequestBody and Timeout directives in your Apache configuration for the Git virtual host or directory.

   # In VirtualHost or Directory block
   LimitRequestBody 0 # 0 for unlimited, or a specific byte size (e.g., 524288000 for 500MB)
   Timeout 300 # Increase timeout
   

   Reload/restart Apache:

   sudo systemctl reload apache2
   

3. Adjust SSH Server Keep-Alive Settings (Server-side)

This is a critical step for repositories accessed via SSH (e.g., [email protected]:user/repo.git). The SSH server might be terminating the connection due to perceived inactivity.

1. Edit sshd_config on the Git server: Connect to your Git server (the machine hosting the repository) via SSH. Open the sshd_config file using a text editor:

   sudo nano /etc/ssh/sshd_config
   

2. Add or modify ClientAliveInterval and ClientAliveCountMax: Add or uncomment and set the following lines. These settings send null packets to the client every ClientAliveInterval seconds, and if no response is received after ClientAliveCountMax intervals, the connection is terminated.

   ClientAliveInterval 60
   ClientAliveCountMax 10
   

   This configuration will send a keep-alive message every 60 seconds and will terminate the connection only after 10 failed attempts (i.e., after 600 seconds or 10 minutes of unresponsiveness). You can increase ClientAliveCountMax for even longer tolerance.

Be mindful when setting ClientAliveInterval to very low values, as it can increase network traffic slightly. However, for preventing disconnects during large transfers, it's often necessary.

3. Restart the SSH service: After saving sshd_config, you must restart the SSH daemon for changes to take effect:

   sudo systemctl restart sshd
   

4. Optimize Git Pack Settings (Client and Server)

For extremely large repositories or those with deep histories, Git's packing process itself can become resource-intensive. Adjusting these settings can reduce memory usage and potentially prevent timeouts.

1. pack.windowMemory: This limits the amount of memory Git uses when searching for delta candidates during packing. Lowering this can reduce peak memory usage but might result in a larger packfile.

   • Client-side:

     git config --global pack.windowMemory 128m # Example: 128 MB
     

   • Server-side (if Git server is doing the packing, e.g., on git clone/git fetch):

     git config --global pack.windowMemory 128m
     

2. pack.depth: This controls how far back Git looks for delta compression. A smaller depth results in faster packing but a potentially larger packfile. For pushes, this usually applies to the client, but the server's configuration might influence fetches/clones.

   • Client-side:

     git config --global pack.depth 50 # Default is 250, try lower
     

These settings might trade off packfile size and network transfer time for reduced memory usage and faster packing. Experiment to find a balance.

5. Verify Server Resources & Limits

Resource exhaustion on the Git server can cause processes to fail, indirectly leading to a SIGPIPE on the client.

1. Check Disk Space: Ensure the server has ample disk space for temporary files and the repository itself.

   df -h
   

2. Check Memory Usage: Monitor memory usage during the push operation. pack-objects can be a memory hog. If the server runs out of RAM, the Linux OOM killer might terminate processes.

   free -h
   

   You can also check dmesg for OOM killer messages:

   dmesg | grep -i oom-killer
   

3. Check ulimit settings: Ensure the user account running the Git server processes (or sshd processes) doesn't have restrictive ulimit settings for open files, memory, or CPU time. These are typically defined in /etc/security/limits.conf or inherited from systemd service configurations.

6. Network Check & MTU

Intermittent network issues, packet loss, or MTU mismatches can also cause connection breaks.

1. Ping and Traceroute: From the client, ping the Git server to check for general connectivity and latency. Use mtr for more detailed path analysis.

   ping -c 10 your.git.server.com
   sudo apt install mtr
   mtr -rw your.git.server.com
   

2. MTU (Maximum Transmission Unit): While less common, an MTU mismatch along the network path can cause packet fragmentation and reassembly issues, leading to connection instability. You might need to adjust the MTU on your network interface, but this should only be done if other network diagnostics strongly suggest it.

   • Check current MTU: ip link show eth0 (replace eth0 with your interface)
   • Test MTU path (example): ping -M do -s 1472 your.git.server.com (1472 bytes + 28 bytes IP/ICMP header = 1500 total, for standard Ethernet).

7. Alternative: Use git bundle for Massive One-Off Transfers

For exceptionally large repositories, especially when migrating or seeding, git bundle can be a more robust method as it doesn't rely on a live streaming connection for the entire transfer.

1. On the source machine (client): Create a bundle file containing all branches or specific refs.

   cd /path/to/source/repo
   git bundle create repo.bundle --all
   

2. Transfer the bundle file: Use scp, rsync, or another reliable file transfer method to move repo.bundle to the target server.

   scp repo.bundle [email protected]:/tmp/
   

3. On the target machine (server): Initialize a new repository and unbundle the contents.

   cd /path/to/destination/on/server
   git init --bare new-repo.git
   cd new-repo.git
   git bundle unbundle /tmp/repo.bundle
   

   Alternatively, to clone directly from a bundle:

   git clone /tmp/repo.bundle /path/to/destination/repo
   

By systematically working through these solutions, you should be able to identify and resolve the underlying cause of the "pack-objects died of signal 13 pipe break" error during your large Git pushes.