Troubleshooting Nginx 504 Gateway Timeout on CentOS Stream / Rocky Linux

Resolve Nginx 504 Gateway Timeout errors on CentOS Stream/Rocky Linux by diagnosing upstream application, PHP-FPM, and Nginx configuration.

Resolve Nginx 504 Gateway Timeout errors on CentOS Stream/Rocky Linux by diagnosing upstream application, PHP-FPM, and Nginx configuration.

A 504 Gateway Timeout error in Nginx indicates that Nginx, acting as a reverse proxy or gateway, did not receive a timely response from an upstream server. This often means your web application (e.g., PHP-FPM, Gunicorn, Node.js, Tomcat) took too long to process a request, exceeding Nginx's configured timeout limits. While Nginx reports the error, the root cause usually lies with the backend application or the communication path to it. This guide provides a comprehensive approach to diagnose and resolve these issues on CentOS Stream or Rocky Linux environments.

Symptom & Error Signature

Users accessing your website will typically see a generic Nginx 504 Gateway Timeout page in their browser:

<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.24.0</center>
</body>
</html>

More critically, the Nginx error log (typically /var/log/nginx/error.log on CentOS/Rocky) will contain specific entries detailing the timeout:

2023/10/27 14:35:01 [error] 12345#12345: *123456 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 203.0.113.1, server: yourdomain.com, request: "GET /long-running-script.php HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm/www.sock:", host: "yourdomain.com"

or for generic proxy setups:

2023/10/27 14:35:01 [error] 12345#12345: *123456 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 203.0.113.1, server: yourdomain.com, request: "GET /api/v1/data HTTP/1.1", upstream: "http://127.0.0.1:8000/api/v1/data", host: "yourdomain.com"

Root Cause Analysis

A 504 Gateway Timeout signifies a breakdown in the communication chain between Nginx and its designated "upstream" server or process. Nginx sent the request, but the upstream didn't respond with a full HTTP response within the configured timeframe.

Common underlying causes include:

  1. Application Processing Time: The most frequent cause. The backend application (e.g., PHP script, Python/Node.js app handler) takes too long to process the request due to:
    • Complex computations.
    • Inefficient database queries (missing indexes, large joins).
    • Slow external API calls or network latency to third-party services.
    • Large file operations.
    • Application bugs leading to infinite loops or resource hogs.
  2. Upstream Service Issues:
    • PHP-FPM: The PHP-FPM process manager might have its own request_terminate_timeout set lower than Nginx's timeout, or it might be overloaded/stuck.
    • Application Server: The upstream application server (e.g., Gunicorn, uWSGI, Tomcat) might be crashed, frozen, or simply overwhelmed and unable to accept new connections or process existing ones efficiently.
  3. Resource Exhaustion: The server hosting the upstream application might be running out of:
    • CPU: High CPU usage can lead to slow processing.
    • Memory (RAM): Swapping to disk drastically slows down processes.
    • I/O: Disk I/O bottlenecks can delay reading/writing data.
    • Network: Network issues between Nginx and the upstream (if they are on different hosts, or even local socket issues).
  4. Nginx Timeout Configuration: While Nginx reports the error, its own timeout directives (proxy_read_timeout, fastcgi_read_timeout, etc.) might be set too low for the expected application load or processing time.
  5. DNS Resolution Issues: If the upstream is specified by a hostname, DNS resolution failures or delays can prevent Nginx from connecting.

Step-by-Step Resolution

Follow these steps to diagnose and resolve the 504 Gateway Timeout.

1. Identify the Upstream Process and Configuration

First, determine which upstream Nginx is trying to communicate with. Open your Nginx configuration files, typically located in /etc/nginx/nginx.conf or /etc/nginx/conf.d/*.conf, and examine the location blocks related to the problematic URL.

Look for directives like proxy_pass or fastcgi_pass:

Example for PHP-FPM (FastCGI):

location ~ .php$ {
    try_files $uri =404;
    fastcgi_pass unix:/run/php-fpm/www.sock; # This is your upstream!
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    include fastcgi_params;
    # Potential timeout directives:
    # fastcgi_read_timeout 300s;
}

Example for a generic HTTP proxy (e.g., Node.js, Python/Gunicorn):

location /api/ {
    proxy_pass http://127.0.0.1:8000; # This is your upstream!
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    # Potential timeout directives:
    # proxy_read_timeout 300s;
}

Make a note of the upstream address (e.g., unix:/run/php-fpm/www.sock or http://127.0.0.1:8000).

2. Check Upstream Service Status & Logs

The next crucial step is to examine the health and logs of your identified upstream service.

For PHP-FPM:

  1. Check PHP-FPM Status:

    systemctl status php-fpm

    Ensure it's active (running). If it's failed or inactive, restart it: systemctl restart php-fpm.

  2. Check PHP-FPM Logs: PHP-FPM logs can reveal errors or processes exceeding their request_terminate_timeout.

    journalctl -u php-fpm -e
# Or check specific PHP-FPM pool logs, often in /var/log/php-fpm/
# E.g., tail -f /var/log/php-fpm/www-error.log

    Look for entries like "script terminated by signal 9 (Kill)" or "maximum execution time exceeded".

For generic HTTP proxy (Node.js, Python, Java, etc.):

  1. Check Application Service Status: If your application runs as a systemd service:

    systemctl status <your-app-service-name>

    Replace <your-app-service-name> with the actual name (e.g., gunicorn, nodeapp, tomcat). Ensure it's running.

  2. Check Application Logs: Most applications log to specific files or via journalctl.

    journalctl -u <your-app-service-name> -e
# Or check app-specific logs, e.g., /var/log/gunicorn/error.log

    Look for application errors, stack traces, or indications of long-running operations.

The upstream application logs are critical for understanding why the process is taking too long. This is often where the root cause is found.

3. Increase Nginx Timeout Directives

If your upstream application is legitimately taking a long time (and you've verified it's not crashing), you might need to increase Nginx's waiting time. Be cautious not to set these values excessively high, as it can hide underlying performance problems.

Add or adjust the following directives within your Nginx configuration. They can be placed in the http, server, or location block depending on the scope required.

For FastCGI (PHP-FPM):

# In http, server, or location block
fastcgi_read_timeout 180s; # Default is 60s
fastcgi_send_timeout 180s; # Default is 60s
fastcgi_connect_timeout 180s; # Default is 60s

fastcgi_read_timeout is the most common one to adjust for 504s with PHP-FPM.

For generic HTTP proxy:

# In http, server, or location block
proxy_read_timeout 180s; # Default is 60s
proxy_send_timeout 180s; # Default is 60s
proxy_connect_timeout 180s; # Default is 60s

proxy_read_timeout is the most common one to adjust for 504s with proxied applications.

Other general timeouts that might affect specific scenarios:

# In http or server block
client_body_timeout 60s; # Default is 60s. For client sending large request body.
client_header_timeout 60s; # Default is 60s. For client sending headers.
send_timeout 60s; # Default is 60s. For Nginx sending response to client.

After modifying Nginx configuration, always test and reload:

nginx -t
systemctl reload nginx

While increasing timeouts can resolve the immediate 504 error, it does not fix the underlying performance issue. Indiscriminately setting very high timeouts (e.g., 600s or more) can lead to requests hanging indefinitely, consuming server resources, and potentially causing other issues. Use judiciously and always aim to optimize the application itself.

4. Optimize Upstream Application Performance

This is often the ultimate solution. A well-performing application won't hit timeout limits.

  • Code Review & Profiling: Identify slow parts of your application code. Use profiling tools specific to your language (e.g., Xdebug for PHP, cProfile for Python, Node.js Inspector).
  • Database Optimization:
    • Add/optimize database indexes.
    • Rewrite inefficient queries.
    • Avoid N+1 queries.
    • Consider database caching.
  • Caching: Implement application-level caching (e.g., Redis, Memcached) for frequently accessed data or computationally expensive results.
  • Asynchronous Processing: For very long-running tasks, consider offloading them to background jobs (e.g., using message queues like RabbitMQ, Kafka, or task queues like Celery for Python) instead of processing them synchronously during a web request.
  • External Service Optimizations: If external API calls are slow, consider caching responses, implementing retries with exponential backoff, or optimizing the external service calls themselves.

5. Adjust Upstream Process Manager Settings (e.g., PHP-FPM)

If PHP-FPM is your upstream, its own configuration can contribute to 504s.

Open your PHP-FPM pool configuration file (e.g., /etc/php-fpm.d/www.conf or a custom pool file).

  1. request_terminate_timeout: This is PHP-FPM's internal timeout for a single script. If it's shorter than Nginx's fastcgi_read_timeout, PHP-FPM will kill the script before Nginx times out, potentially leading to a 502 Bad Gateway or a script error rather than a 504. Adjust it to be at least equal to or slightly higher than fastcgi_read_timeout.

    ; In /etc/php-fpm.d/www.conf (or your pool config)
request_terminate_timeout = 180s

  2. PHP max_execution_time: In php.ini, ensure max_execution_time is also sufficiently high.

    ; In /etc/php.ini
max_execution_time = 180

  3. PHP-FPM Process Management: If PHP-FPM pools are exhausted, new requests will queue or fail.

    • pm = dynamic or pm = ondemand: These are common.
    • pm.max_children: The maximum number of PHP-FPM processes. Increase if your server has enough RAM.
    • pm.start_servers, pm.min_spare_servers, pm.max_spare_servers: Tune these to ensure a sufficient number of idle child processes are available to handle incoming requests without delays.
    • pm.max_requests: The number of requests each child process will execute before respawning. Can prevent memory leaks in long-running processes but respawning can momentarily reduce available children.

After modifying PHP-FPM configuration, restart the service:

systemctl restart php-fpm

Incorrect PHP-FPM tuning (especially pm.max_children) can lead to excessive memory consumption and server instability. Monitor your RAM usage (free -h) carefully after changes. A good starting point is (Total RAM - OS_RAM - DB_RAM) / (Avg PHP Process RAM).

6. System Resource Monitoring

Monitor your server's resources during periods of high load or when the 504 errors occur.

  • CPU Usage:
    top # or htop (if installed)
    Look for high us (user CPU) or wa (I/O wait) percentages.
  • Memory Usage:
    free -h
    High Swap usage indicates memory exhaustion.
  • Disk I/O:
    iostat -x 1 5 # install with 'dnf install sysstat'
    High %util and avgqu-sz can point to disk bottlenecks.
  • Network Activity:
    ss -s # Summarizes socket statistics
netstat -antp # Shows active TCP connections
    Look for an excessive number of connections or high retransmits.

If you find resource bottlenecks, consider:

  • Upgrading server hardware (CPU, RAM, faster storage).
  • Optimizing your application to use fewer resources.
  • Scaling horizontally by adding more application servers behind a load balancer.

7. Advanced Troubleshooting & Tracing

For persistent and complex issues, more advanced techniques might be necessary:

  • strace: For an extremely specific look at what a process is doing, strace can trace system calls. Use with caution on production as it can significantly slow down the traced process.
    # Example: trace a specific PHP-FPM child process PID
strace -p <PID_OF_PHP_FPM_CHILD>
  • Nginx Stub Status Module: Enable the Nginx ngx_http_stub_status_module to monitor Nginx's own connections and requests per second.
  • Distributed Tracing: For microservices architectures or complex applications, implement distributed tracing (e.g., OpenTelemetry, Jaeger, Zipkin) to visualize the flow of requests and pinpoint bottlenecks across services.

8. Reload Nginx and Upstream Services

After making any configuration changes, always test and reload/restart the affected services.

  1. Test Nginx Configuration:

    nginx -t

    This command checks for syntax errors in your Nginx configuration files. If there are no errors, proceed.

  2. Reload Nginx:

    systemctl reload nginx

    A reload applies new configurations without dropping active connections.

  3. Restart PHP-FPM (or your upstream service):

    systemctl restart php-fpm
# Or for generic app:
systemctl restart <your-app-service-name>

    Use restart for PHP-FPM if you've changed pm settings or request_terminate_timeout. For other applications, restart is generally safer after significant config changes. If the service supports reload, that's preferred to avoid downtime.

By systematically working through these steps, from diagnosing the upstream to optimizing application performance and tuning server resources, you can effectively resolve Nginx 504 Gateway Timeout errors on your CentOS Stream or Rocky Linux system.