Troubleshooting Git `pack-objects died of signal 13` Pipe Break on CentOS/Rocky Linux: Large Push Fix

Resolve Git push failures with 'pack-objects died of signal 13 pipe break' on CentOS Stream or Rocky Linux. This guide covers server-side resource limits, Git config, and SSH timeouts for large repositories.

When working with large Git repositories, especially during initial pushes or pushes containing significant binary changes, you might encounter a cryptic error message indicating that git pack-objects died. This usually signifies a resource exhaustion issue on the Git server, leading to a broken pipe during the data transfer. This guide delves into the root causes and provides comprehensive, step-by-step solutions for CentOS Stream and Rocky Linux environments.

Symptom & Error Signature

Users experience failed git push operations, often after a long period of transfer activity, resulting in messages similar to these:

Enumerating objects: 1234567, done.
Counting objects: 100% (1234567/1234567), done.
Delta compression using up to 16 threads
Compressing objects: 100% (1234560/1234560), done.
Writing objects: 100% (1234567/1234567), 4.88 GiB | 12.00 MiB/s, done.
Total 1234567 (delta 987654), reused 1234567 (delta 987654), pack-reused 0
error: remote unpack failed: unpack-objects died with exit code 1
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly

Alternatively, you might see a more explicit Broken pipe or signal 13 error:

Counting objects: 1234567, done.
Delta compression using up to 8 threads
Compressing objects: 100% (1234560/1234560), done.
Writing objects: 100% (1234567/1234567), 4.88 GiB | 12.00 MiB/s
fatal: pack-objects died of signal 13
fatal: The remote end hung up unexpectedly

Root Cause Analysis

The error pack-objects died of signal 13 (SIGPIPE) indicates that the git pack-objects process on the server-side, responsible for compressing and packing repository objects into an efficient format for transfer, was terminated prematurely because the pipe it was writing to was closed by the reading end. This closure typically happens due to underlying resource constraints or timeouts on the server or network.

Here's a breakdown of common root causes:

1. Server-Side Resource Exhaustion:

   • Memory (OOM Killer): git pack-objects can be highly memory-intensive, especially for large repositories with deep histories or many large binary files. If the server runs out of available RAM, the Linux Out-Of-Memory (OOM) killer will terminate processes, often targeting pack-objects due to its high consumption. This is the most common cause.
   • CPU Time Limits: Less frequent, but if a system has strict CPU time limits for user processes or services, a long-running pack-objects operation might exceed them.
   • Temporary Disk Space: Creating large pack files requires sufficient temporary disk space. If /tmp or the Git repository's temporary directory runs out of space, pack-objects can fail.
   • File Descriptors: While less direct for SIGPIPE, insufficient file descriptors could indirectly contribute to process instability.

2. SSH/HTTP Timeouts:

   • If you're pushing via SSH, the sshd daemon on the server might have strict ClientAliveInterval or ClientAliveCountMax settings, causing the SSH session to terminate if no "keep-alive" data is received for an extended period, leading to a broken pipe.
   • Similar timeouts can occur with HTTP/HTTPS proxies or web servers (e.g., Nginx serving Git over HTTP).

3. Git Configuration Limits:

   • receive.maxInputBufferSize: This server-side Git configuration setting limits the total size of data that Git will accept during a push operation before unpacking. If your push exceeds this limit, the process might be terminated. The default is usually 1GB.

4. Network Instability:

   • While less likely to directly cause a SIGPIPE from pack-objects, intermittent network connectivity issues or highly latent connections can lead to dropped SSH/HTTP connections, which then results in the server-side pipe breaking.

Step-by-Step Resolution

Addressing this issue primarily involves increasing server-side resource limits and adjusting Git/SSH configurations.

1. Analyze Server-Side Logs for Clues

Before making changes, check server logs for specific errors.

• OOM Killer Logs: Look for Out of memory or killed process messages in journalctl.

  sudo journalctl -b | grep -i "oom-killer|killed process"
  

  This will often show which process was killed and why.

• SSH Daemon Logs: Check sshd logs for connection terminations.

  sudo journalctl -u sshd | grep -i "Disconnected|timeout"
  

• Git User Logs (if applicable): If your Git server runs under a dedicated user (e.g., git user for bare repos via SSH), check its ~/.bash_history for any unusual commands or its home directory for specific Git log files (though pack-objects typically logs to stderr/stdout or system logs).

2. Increase Server-Side Resource Limits

This is often the most critical step. We'll focus on memory and file descriptor limits.

A. Increase System-wide and sshd PAM Limits

Edit /etc/security/limits.conf to increase limits for the user under which git pack-objects runs (often the git user or the user account that SSH's into to manage repositories).

sudo vi /etc/security/limits.conf

Add or modify the following lines (replace gituser with your actual Git user if different):

# Git user resource limits for large pushes
gituser hard nofile 65535  # Max open files
gituser soft nofile 65535
gituser hard nproc  65535  # Max processes
gituser soft nproc  65535
gituser hard as    unlimited # Max address space (virtual memory)
gituser soft as    unlimited
gituser hard rss   unlimited # Max resident set size (physical memory)
gituser soft rss   unlimited

The as (address space) and rss (resident set size) limits are crucial for memory-intensive operations. Setting them to unlimited removes any artificial memory caps imposed by PAM, allowing the process to use available system RAM.

For these changes to take effect, the user must log out and log back in, or for sshd, the sshd service might need a restart (though typically PAM limits apply on new sessions).

B. Adjust sshd Service Limits (Systemd)

Sometimes, the sshd service itself or the processes it spawns might be subject to Systemd unit limits. First, check the current limits for sshd:

systemctl show sshd | grep -i "Limit"

If you see restrictive limits like LimitAS, LimitRSS, LimitNOFILE, etc., you can override them. Create a drop-in unit file for sshd:

sudo systemctl edit sshd.service

This will open an editor (e.g., vi or nano) for /etc/systemd/system/sshd.service.d/override.conf. Add the following:

[Service]
LimitMEMLOCK=infinity
LimitNOFILE=65535
LimitNPROC=65535
LimitAS=infinity
LimitRSS=infinity

Save and exit. Then reload Systemd and restart sshd:

sudo systemctl daemon-reload
sudo systemctl restart sshd

C. Increase System-wide Default Limits (Systemd)

For general processes, you can also modify /etc/systemd/system.conf and /etc/systemd/user.conf to increase default limits, though PAM limits are usually more direct for the git user.

sudo vi /etc/systemd/system.conf
sudo vi /etc/systemd/user.conf

Uncomment and set higher values for:

DefaultLimitNOFILE=65535
DefaultLimitNPROC=65535
DefaultLimitAS=infinity
DefaultLimitRSS=infinity

Apply changes:

sudo systemctl daemon-reexec

3. Adjust Git Server Configuration

A. Increase receive.maxInputBufferSize

This Git server-side configuration controls the maximum size of a push that Git will process before unpacking. Increase it to accommodate very large pushes. This can be done globally or per-repository.

Globally (for all repositories on the server for the user):

# As the gituser
git config --global receive.maxInputBufferSize 5242880000 # 5GB (in bytes)

Per Repository (in a bare repo):

# Inside the bare repository, as the gituser
git config receive.maxInputBufferSize 5242880000 # 5GB (in bytes)

The value is in bytes. 5GB or even 10GB might be necessary for extremely large pushes.

For HTTP-based Git servers, you might also need to adjust http.postBuffer on the client side, but for pack-objects died of signal 13, the server-side receive.maxInputBufferSize is usually more relevant.

4. Adjust SSH Daemon Timeouts (Server-Side)

If SSH timeouts are a factor, increase them.

sudo vi /etc/ssh/sshd_config

Modify or add these lines:

ClientAliveInterval 300 # Sends a null packet to the client every 300 seconds (5 minutes)
ClientAliveCountMax 12  # Allows for 12 unresponsive intervals before disconnecting (300 * 12 = 3600 seconds or 1 hour)

A ClientAliveInterval of 300 and ClientAliveCountMax of 12 would mean the SSH session can remain idle/unresponsive from the client's perspective for up to 1 hour before being forcibly disconnected. Adjust these values based on your network stability and expected push times.

After modifying, restart the sshd service:

sudo systemctl restart sshd

5. Verify Disk Space

Ensure there's ample free space on the partition hosting the Git repositories and on /tmp (where temporary pack files might be created).

df -h /path/to/git/repos /tmp

If disk space is low, free up space or expand the filesystem.

6. Optimize the Repository (Preventative/Cleanup)

While these won't directly fix an ongoing pack-objects failure, they are good practices for large repositories and can prevent future issues.

A. Run git gc on the server

If the server-side repository has accumulated many loose objects, git gc can clean and repack them, making future operations more efficient.

# As the gituser, inside the bare repository
git gc --aggressive --prune=now

B. Consider Git LFS for Large Binary Files

If your repository contains many large binary files (images, videos, executables, large datasets), Git LFS (Large File Storage) is the recommended solution. Git LFS replaces large files with small pointer files in the Git repository, storing the actual binary content outside the Git repo. This dramatically reduces repository size and transfer times.

Implementing Git LFS for existing large files requires rewriting history (git filter-repo or similar), which can be complex and disruptive. It's best used for new large files or for cleaning up history if absolutely necessary.

7. Client-Side Considerations

While the error usually points to the server, ensure your client-side environment is optimal.

A. Increase HTTP Post Buffer (if using HTTP/HTTPS)

If you are pushing via HTTP/HTTPS, the client's http.postBuffer might need to be increased. This is the buffer Git uses for HTTP POST requests.

git config --global http.postBuffer 524288000 # 500MB

B. Update Git Client

Ensure your Git client is up-to-date. Newer Git versions often have performance improvements.

# On CentOS/Rocky Linux client
sudo dnf update git

8. Try a Shallow Clone/Push (Workaround)

If all else fails for an initial large clone, or if you can segment your push:

• Shallow Clone: For clients, a shallow clone reduces the amount of history downloaded. Not directly a push fix, but useful for client-side performance.

  git clone --depth 1 <repository_url>
  

• Push in Chunks: If you have many branches or tags, try pushing them individually rather than all at once. For an initial large push, if you can squash commits locally, pushing fewer, larger commits might sometimes bypass issues, but this alters history.

By systematically applying these solutions, starting with server-side resource limits and Git configuration, you should be able to resolve the pack-objects died of signal 13 error on your CentOS Stream or Rocky Linux Git server and successfully push your large repositories.