Troubleshooting Git Error: ‘pack-objects died of signal 13 pipe break’ on Large Pushes (Ubuntu 22.04 LTS)

Resolve 'pack-objects died of signal 13 pipe break' during large Git pushes on Ubuntu 22.04. This guide covers common causes and solutions for SIGPIPE errors, including HTTP and SSH configurations.

A "Git error pack-objects died of signal 13 pipe break" often surfaces when attempting to push a large volume of data to a remote Git repository. This error indicates that the git-pack-objects process, responsible for packaging objects for transfer, prematurely terminated due to a broken pipe (SIGPIPE). This guide provides a detailed, step-by-step approach to diagnose and resolve this complex issue on Ubuntu 22.04 LTS systems.

Symptom & Error Signature

When pushing a particularly large repository, or one containing significant changes, your Git client may display output similar to the following, often after a long period of "Compressing objects" or "Writing objects":

Typical HTTP/HTTPS Push Error:

Enumerating objects: 123456, done.
Counting objects: 100% (123456/123456), done.
Delta compression using up to 16 threads
Compressing objects: 100% (123450/123450), 25.00 GiB | 12.34 MiB/s, done.
Writing objects: 100% (123456/123456), 25.00 GiB | 12.34 MiB/s, done.
Total 123456 (delta 98765), reused 123456 (delta 98765), pack-reused 0
error: remote unpack failed: unable to create temporary file: No space left on device
error: remote unpack failed: error processing pack: pack-objects died of signal 13
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly

Typical SSH Push Error:

Enumerating objects: 123456, done.
Counting objects: 100% (123456/123456), done.
Delta compression using up to 16 threads
Compressing objects: 100% (123450/123450), 25.00 GiB | 12.34 MiB/s, done.
Writing objects: 100% (123456/123456), 25.00 GiB | 12.34 MiB/s, done.
Total 123456 (delta 98765), reused 123456 (delta 98765), pack-reused 0
fatal: The remote end hung up unexpectedly
fatal: pack-objects died of signal 13

The key indicators are pack-objects died of signal 13 and fatal: The remote end hung up unexpectedly. The No space left on device message is a common accompanying symptom if the server-side disk is full.

Root Cause Analysis

The error pack-objects died of signal 13 refers to SIGPIPE, a signal sent to a process when it attempts to write to a pipe whose reading end has been closed. In the context of Git, this usually means the connection to the remote Git server (or an intermediate process) was unexpectedly severed while the git-pack-objects process was still trying to send data.

The underlying causes are typically server-side, though client-side network issues or misconfigurations can also contribute:

• Server-Side Resource Exhaustion:

  • Disk Space: The most frequent culprit. The remote server, especially in the temporary directory used for unpacking the incoming pack, runs out of disk space. Git cannot write the temporary pack files, causing the process to fail and the pipe to close.
  • Memory (RAM): The git-receive-pack or git-upload-pack process on the server, which handles the incoming data, might hit system-defined memory limits (ulimit -v, ulimit -d, LimitAS, LimitRSS). Processing large packs requires significant RAM.
  • CPU/I/O Load: An overloaded server or slow disk I/O can cause processes to timeout or be killed by the OOM (Out Of Memory) killer if memory pressure is severe.

• Server-Side Timeouts:

  • HTTP/HTTPS: If Git is served via HTTP(S) (e.g., smart HTTP), web server configurations (Nginx, Apache) often have strict timeouts (proxy_read_timeout, send_timeout, client_body_timeout) and upload size limits (client_max_body_size) that can cut off long-running pushes.
  • SSH: SSH daemon (sshd) configurations might terminate inactive sessions (ClientAliveInterval, ClientAliveCountMax) or aggressively limit concurrent connections (MaxStartups, MaxSessions), though less common for single large pushes.

• Server-Side Git Hooks:

  • pre-receive or update hooks configured on the remote repository can fail, timeout, or exit with an error code, causing the remote Git process to terminate prematurely and break the pipe. This is especially true if hooks perform extensive checks or operations on the incoming data.

• Network Instability/Intermediate Proxies:

  • Unstable network connections, VPNs, or intermediate proxies/firewalls might have their own connection limits or timeouts that interrupt the long data transfer.

• ulimit Restrictions:

  • System-wide or user-specific resource limits (ulimit) on the server for the user running the Git processes (e.g., git user, or web server user like www-data for HTTP Git) can restrict memory, file size, or process time, leading to pack-objects failure.

Step-by-Step Resolution

Address these potential causes systematically, starting with the most common and easiest to check.

1. Verify Server-Side Disk Space

The single most common cause for "pack-objects died" is insufficient disk space on the remote server where the repository resides. Git needs free space for temporary files during the unpacking process.

# On the server machine
df -h

Check the output for the partition where your Git repositories are stored (e.g., /var/www/git, /home/git). If it shows 100% or very high utilization, you need to free up space or extend the partition.

Git might also use /tmp or a user's home directory for temporary files. Ensure these also have ample free space.

2. Increase Git Client-Side HTTP Buffer (for HTTP/HTTPS Pushes)

For large pushes over HTTP/HTTPS, the client might not allocate enough buffer space to send the large pack file, leading to disconnects.

# On the client machine
# Increase the HTTP post buffer globally (for all repositories)
git config --global http.postBuffer 524288000 # Set to 500 MB (500 * 1024 * 1024 bytes)

# Or for a specific repository (from within the repository directory)
# git config http.postBuffer 524288000

Start with 500 MB. For extremely large repositories (tens of GBs), you might need to increase this to 1G (1073741824) or 2G.

3. Adjust Nginx Configuration (for HTTP/HTTPS Git Servers)

If your Git repository is served via Nginx (e.g., Git smart HTTP, GitLab, Gitea), you'll need to adjust Nginx's limits to allow large file uploads and long connection times.

# On the server machine, edit your Nginx configuration.
# Common paths: /etc/nginx/nginx.conf, /etc/nginx/sites-available/your-git-site.conf

# 1. Increase client_max_body_size in http block or server block
# Allows for large request bodies (the Git pack)
http {
    # ...
    client_max_body_size 0; # Allows unlimited body size. Alternatively, set a large specific value like 10G.
    # ...
}

# 2. Increase timeouts in your server/location block for the Git service
server {
    listen 443 ssl;
    server_name git.example.com;

    location / {
        # ...
        proxy_read_timeout 3600; # 1 hour
        proxy_send_timeout 3600; # 1 hour
        send_timeout 3600;       # 1 hour
        # Optional: If you use FastCGI (e.g., fcgiwrap for Gitweb/cgit)
        # fastcgi_read_timeout 3600;
        # fastcgi_send_timeout 3600;
        # ...
    }
}

After modifying Nginx configuration, always test the configuration for syntax errors and then reload/restart Nginx:

sudo nginx -t
sudo systemctl reload nginx

4. Review SSH Server Configuration (for SSH Git Servers)

For pushes over SSH, sshd timeouts can be a factor, though less common for breaking pipes specifically during pack-objects.

# On the server machine, edit the SSH daemon configuration
sudo vim /etc/ssh/sshd_config

Look for or add the following directives:

# /etc/ssh/sshd_config
# ...
ClientAliveInterval 300   # Send a null packet to the client every 300 seconds (5 minutes)
ClientAliveCountMax 12    # Number of client alive messages without a response.
                          # This would mean a timeout after 300 * 12 = 3600 seconds (1 hour) of client inactivity.
                          # Increase ClientAliveCountMax if you have very long pushes that might appear 'inactive'
                          # at times to the SSH server, e.g., during complex server-side hook processing.

# Optional: To increase overall concurrent connections, if hitting limits (less common for pipe break)
# MaxStartups 10:30:60 # Max unauthenticated connections (start:rate:full)
# MaxSessions 10      # Max open sessions per network connection
# ...

After modifying sshd_config, you must restart the SSH service for changes to take effect:

sudo systemctl restart sshd

5. Increase Server-Side ulimit for Git User/Process

Git processes, especially git-receive-pack and git-unpack-objects, can consume significant memory and other resources. System-level ulimit restrictions might kill these processes.

First, identify the user under which Git operations are performed on the server. This could be a dedicated git user, the web server user (www-data for HTTP Git), or a service user (e.g., git for GitLab, gitea for Gitea).

Check current limits for the relevant user:

# Example for a 'git' user
sudo su - git -c "ulimit -a"

# Example for 'www-data' user
sudo su - www-data -s /bin/bash -c "ulimit -a"

Look for as (address space), data (data segment size), and rss (resident set size) limits. If they are not unlimited or sufficiently high, they might be the problem.

Modify limits.conf (for login shells/system-wide):

sudo vim /etc/security/limits.conf

Add or modify entries for your Git user/group:

# /etc/security/limits.conf
# ...
# <domain>      <type>  <item>         <value>
git             soft    as             unlimited # Address space (virtual memory)
git             hard    as             unlimited
git             soft    data           unlimited # Data segment size
git             hard    data           unlimited
git             soft    rss            unlimited # Resident set size (physical memory)
git             hard    rss            unlimited
# ...

Changes in limits.conf require the user to log out and log back in, or for the service (if any) to be restarted to pick up the new limits. Setting unlimited for critical resources should be done with caution, understanding your server's capabilities.

Modify Systemd Service Unit Files (for Git services like Gitea/GitLab):

If Git is run by a Systemd service (common for self-hosted solutions like Gitea, GitLab, or custom Git daemons), you can set limits directly in the service's unit file or an override.

# Example for Gitea service
sudo systemctl edit gitea.service

Add or modify the [Service] section in the override file:

# /etc/systemd/system/gitea.service.d/override.conf
[Service]
LimitAS=infinity   # Address space (virtual memory)
LimitDATA=infinity # Data segment size
LimitRSS=infinity  # Resident set size (physical memory)
# You can also set specific values, e.g., LimitAS=8G (8 gigabytes)

After modifying a Systemd service file, reload the daemon and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart gitea

6. Debug and Optimize Git Server-Side Hooks

If your remote repository uses pre-receive or update hooks, these scripts execute before the push is fully accepted. A failing or excessively slow hook can cause the remote Git process to terminate and break the pipe.

1. Temporarily Disable Hooks: Navigate to the bare repository's hooks directory on the server:

   cd /path/to/your/repo.git/hooks
   sudo mv pre-receive pre-receive.bak # Temporarily disable
   sudo mv update update.bak           # Temporarily disable if present
   

   Attempt the push again. If it succeeds, the issue is within your hooks.

2. Add Logging to Hooks: If disabling hooks resolves the issue, restore them (mv pre-receive.bak pre-receive) and add verbose logging to debug which part is failing or timing out.

   #!/bin/bash
   # Add this to the top of your hook script (e.g., pre-receive)
   LOGFILE="/tmp/git-hook-debug-$(date +%Y%m%d%H%M%S).log"
   exec > "$LOGFILE" 2>&1
   echo "Hook started by user $USER at $(date)"
   echo "Received refs: $@"
   # ... your original hook logic ...
   echo "Hook finished at $(date)"
   exit 0 # Ensure your hook exits with 0 on success
   

   Then, inspect the log file (/tmp/git-hook-debug-*.log) after a failed push.

7. Consider Alternative Push Methods or Repository Restructuring

If your repository is genuinely enormous (multiple gigabytes, especially with large binary files) and the above solutions don't help, it might be a structural issue.

• Git LFS (Large File Storage): For large binary files (images, videos, archives, executables) that bloat your Git repository history, use Git LFS. It stores large files outside the Git repository while keeping pointers in Git.

  # On the client machine
  git lfs install
  git lfs track "*.zip" # Or other large file types
  git add .gitattributes
  git commit -m "Configure Git LFS for zip files"
  git push
  

  This requires a Git LFS server component on the remote.

• Repository Splitting: If the repository contains entirely independent components, consider splitting it into smaller, more manageable repositories.

• Shallow Clones/Pushes: For CI/CD pipelines, consider using shallow clones or limited pushes if the full history isn't needed.

8. Network-Level Troubleshooting

While less direct to pack-objects died, network instability or aggressive firewall rules can cause connection breaks.

• Check Firewall Rules: Ensure no firewalls (UFW, iptables, cloud security groups) on either client or server are prematurely closing connections due to size or duration.
• Test Network Stability: Use ping and mtr to diagnose latency or packet loss between your client and the Git server.

By systematically working through these troubleshooting steps, you should be able to identify and resolve the "Git error pack-objects died of signal 13 pipe break" when performing large pushes on your Ubuntu 22.04 LTS system. Remember to apply changes carefully and test thoroughly after each modification.