Kubernetes Ingress Controller Path Routing Returns 404 Error on CentOS Stream / Rocky Linux

Troubleshoot 404 errors with Kubernetes Ingress path routing on CentOS/Rocky Linux. Diagnose misconfigurations, backend issues, and apply effective fixes.

Troubleshoot 404 errors with Kubernetes Ingress path routing on CentOS/Rocky Linux. Diagnose misconfigurations, backend issues, and apply effective fixes.

When deploying applications to Kubernetes, path-based routing via an Ingress Controller is a common strategy to expose multiple services under a single hostname. However, encountering a "404 Not Found" error specifically for certain paths, while the root path or other services might be working, is a frustrating yet frequent issue. This guide will walk you through diagnosing and resolving these path routing 404s on Kubernetes clusters running on CentOS Stream or Rocky Linux nodes, specifically focusing on the widely used NGINX Ingress Controller.

Symptom & Error Signature

Users attempting to access a specific path on your application (e.g., https://myapp.example.com/api/v1/users) receive a 404 error, while the base URL (https://myapp.example.com) or other paths might load correctly.

Typical Browser Output:

404 Not Found
nginx/1.23.3 (or similar NGINX version)

curl -v Output:

$ curl -v https://myapp.example.com/api/v1/users
*   Trying 203.0.113.5:443...
* Connected to myapp.example.com (203.0.113.5) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
... (SSL/TLS handshake) ...
> GET /api/v1/users HTTP/2
> Host: myapp.example.com
> User-Agent: curl/7.81.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4)
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4)
* old SSL session ID is invalidated
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 404
< date: Mon, 09 Aug 2026 14:30:00 GMT
< content-type: text/html
< content-length: 153
< server: nginx/1.23.3
<
<!DOCTYPE html>
<html>
<head>
<title>404 Not Found</title>
</head>
<body>
<h1>404 Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body>
</html>
* Connection #0 to host myapp.example.com left intact

NGINX Ingress Controller Logs (example):

$ kubectl logs -n ingress-nginx ingress-nginx-controller-abcde-12345
...
2026/08/09 14:30:00 [error] 37#37: *12345 "/usr/share/nginx/html/api/v1/users/index.html" is not found (2: No such file or directory), client: 10.0.0.1, server: myapp.example.com, request: "GET /api/v1/users HTTP/1.1", host: "myapp.example.com"
2026/08/09 14:30:00 [warn] 37#37: *12345 a client request body is buffered to a temporary file /var/lib/nginx/tmp/client_body/0000000001, client: 10.0.0.1, server: myapp.example.com, request: "GET /api/v1/users HTTP/1.1", host: "myapp.example.com"
2026/08/09 14:30:00 "GET /api/v1/users HTTP/1.1" 404 153 "-" "curl/7.81.0" 10.0.0.1:4567 0.002 0.002 [myapp-example-com-service-80] [] 10.244.1.2:8080 153 0.002 404
...

Root Cause Analysis

A 404 error from an Ingress Controller, specifically for a path, usually indicates that the Ingress controller received the request but couldn't find a matching rule to route it to a backend service, or it routed it, but the backend service itself returned a 404.

Here are the most common root causes:

  1. Incorrect Ingress Rule Configuration:

    • Mismatched path or pathType: The path defined in the Ingress resource does not accurately match the incoming request path. The pathType (Prefix, Exact, ImplementationSpecific) determines how paths are matched.
    • Missing rewrite-target annotation: The Ingress receives /api/v1/users but the backend application expects /users or / at its root. Without rewrite-target, the Ingress controller passes the full path to the backend, which then returns a 404.
    • Incorrect serviceName or servicePort: The Ingress rule points to a non-existent service or an incorrect port on the service.

  2. Backend Application Issues:

    • Application not serving on the expected path: The application within your pod might not be configured to listen for requests on the specific /api/v1/users path.
    • Application crash or unhealthy: The backend pods are not running or are in a crashing state, causing the service to have no healthy endpoints.
    • Application returning its own 404: The Ingress controller correctly forwards the request, but the application itself does not have a handler for the requested path and returns a 404.

  3. Kubernetes Service and Endpoint Issues:

    • Service Selector Mismatch: The Kubernetes Service object's selector doesn't match any running Pod labels, resulting in no endpoints for the service.
    • Pod Health Probes Failing: Pods are running but failing their readiness or liveness probes, causing kube-proxy to remove them from the service's endpoint list.

  4. Ingress Controller Issues (Less common for path 404s, more for general connectivity):

    • The NGINX Ingress Controller itself isn't healthy, or its configuration reloads are failing, preventing it from applying new Ingress rules.
    • Resource constraints on the Ingress Controller pod leading to missed updates.

  5. Host-Level Firewall (firewalld) on CentOS Stream / Rocky Linux (Less common for 404, more for connection refused):

    • While less likely to cause a 404 (which implies the request reached the Ingress Controller), if firewalld is misconfigured on your Kubernetes worker nodes, it could potentially block internal cluster communication between the Ingress Controller and the backend service pods, or prevent external traffic from reaching the Ingress Controller's NodePort/LoadBalancer.

Step-by-Step Resolution

Follow these steps to diagnose and resolve Kubernetes Ingress path routing 404 errors.

1. Verify Ingress Resource Definition

The first and most crucial step is to examine your Ingress resource for misconfigurations.

kubectl get ingress <your-ingress-name> -n <your-namespace> -o yaml

Look closely at the rules, host, path, pathType, service.name, service.port.number, and any relevant annotations.

# Example of a problematic Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
  namespace: default
  annotations:
    # nginx.ingress.kubernetes.io/rewrite-target: /$2 # Potentially missing or incorrect
spec:
  ingressClassName: nginx
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /api/v1/users
        pathType: Exact # Or Prefix, check which is appropriate
        backend:
          service:
            name: myapp-service
            port:
              number: 8080

  • pathType:

    • Prefix: Matches URL path by a prefix. /foo would match /foo, /foo/, /foo/bar. This is generally safer.
    • Exact: Matches URL path exactly. /foo would only match /foo, not /foo/bar.
    • ImplementationSpecific: Behavior depends on the IngressClass. NGINX Ingress Controller treats this similarly to Prefix for non-regex paths.
    • If you intend to match /api/v1/users and anything under it, use pathType: Prefix with path: /api/v1/users. If you only want to match that exact path, use pathType: Exact.

  • backend.service.name and backend.service.port.number: Ensure these precisely match your Kubernetes Service name and the targetPort exposed by your application's Service.

The pathType field is critical. A pathType: Exact on /api/v1/users will not match /api/v1/users/profile. Conversely, a pathType: Prefix on /api will match /api/v1/users but might also match other unintended paths. Choose carefully.

2. Inspect Ingress Controller Logs

The Ingress Controller logs often contain valuable clues about why a request wasn't routed correctly.

  1. Find your Ingress Controller pod:

    kubectl get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx
# Output example: ingress-nginx-controller-6d7c7b8c9d-abcdefg

    (Adjust ingress-nginx namespace if yours is different.)

  2. View the logs:

    kubectl logs -n ingress-nginx <ingress-nginx-controller-pod-name> --tail=100

    Look for entries related to the 404 request. Search for the requested host and path. Common messages might include:

    • no route for host <hostname>/<path>
    • no matching ingress rule
    • service <namespace>/<service-name> does not have any active endpoints (this points to backend issues).

3. Validate Kubernetes Service and Endpoints

Even if your Ingress rule is perfect, if the backend service isn't properly exposing your pods, you'll get a 404.

  1. Check your Service definition:

    kubectl get svc myapp-service -n default -o yaml

    Ensure the selector matches the labels of your application pods and that ports are correctly defined.

  2. Verify Service Endpoints:

    kubectl get ep myapp-service -n default

    You should see at least one IP address listed under ENDPOINTS. If myapp-service has <none> listed, it means your service selector isn't finding any healthy pods.

    # Example of healthy endpoints
NAME              ENDPOINTS             AGE
myapp-service     10.244.1.2:8080       2d

    If no endpoints are listed, proceed to step 4.

4. Debug Backend Application and Pods

If the Ingress Controller logs indicate a successful routing attempt, or if your service has no endpoints, the problem lies within your application pods.

  1. List your application pods:

    kubectl get pods -l app=my-app -n default
# Example: my-app-deployment-789abcde-fghij

    Check the STATUS column for Running. If pods are CrashLoopBackOff, Error, or not Running, investigate those pods.

  2. Check application pod logs:

    kubectl logs my-app-deployment-789abcde-fghij -n default

    Look for application-specific errors, startup failures, or messages indicating the application isn't listening on the expected port or path.

  3. Test the application directly inside the pod:

    kubectl exec -it my-app-deployment-789abcde-fghij -n default -- curl -v localhost:8080/api/v1/users

    (Replace 8080 with your application's actual listening port.)

    • If this returns a 200 OK, the problem is likely with the Ingress or Service.
    • If it returns a 404, your application itself is not configured to handle that specific path.

  4. Test the Service directly via port-forward:

    kubectl port-forward svc/myapp-service 8080:8080 -n default
# In another terminal:
curl -v http://localhost:8080/api/v1/users

    This bypasses the Ingress controller entirely and connects directly to your Kubernetes service. If this still returns a 404, the issue is definitely with your application or service configuration.

Testing directly inside the pod or via kubectl port-forward are powerful techniques to isolate whether the 404 is from the Ingress layer or the application backend.

5. Review rewrite-target Annotation

A common cause of path-based 404s is when the backend application expects a different path than what the Ingress receives. For instance, the Ingress might receive /api/v1/users, but your application only understands /users. The nginx.ingress.kubernetes.io/rewrite-target annotation helps with this.

Example scenario:

  • Incoming request path: /api/v1/users
  • Application expects path: /users

Correct Ingress configuration:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  ingressClassName: nginx
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /api/v1/(.*) # Use a regex to capture the rest of the path
        pathType: Prefix
        backend:
          service:
            name: myapp-service
            port:
              number: 8080
  • path: /api/v1/(.*): This regex captures everything after /api/v1/ into a group.
  • nginx.ingress.kubernetes.io/rewrite-target: /$2: The $2 refers to the content captured by the second capturing group in the path regex. In this case, (.*) is the first group (if (/api/v1)(.*) was used, then (.*) would be the second group). For simple cases like this, where /api/v1/ is removed, using /(.*) in the path with $1 as the rewrite target is more common for the part you want to pass. A common pattern to strip a prefix is path: /prefix(/|$)(.*) and rewrite-target: /$2.

When using rewrite-target, ensure your path uses regular expressions properly to capture the desired segments. Consult the NGINX Ingress Controller documentation for detailed examples of rewrite-target usage. Incorrect regex or group references ($1, $2, etc.) will lead to incorrect path rewriting.

6. Check CentOS/Rocky Linux Firewall (firewalld)

While less common for a 404 (as the request usually reaches the Ingress Controller), if the cluster's networking or kube-proxy is misbehaving, or if firewalld is explicitly blocking traffic to NodePorts or internal cluster communication, it could manifest as a routing failure or timeout.

Modifying firewalld on Kubernetes nodes without understanding its implications can disrupt cluster communication and break your deployment. Proceed with caution. Kubernetes usually manages iptables rules, but firewalld can interfere if not properly configured to allow iptables to take precedence or if critical ports are explicitly blocked.

  1. Check firewalld status on your worker nodes (where Ingress Controller or backend pods run):

    sudo systemctl status firewalld

  2. List open ports and zones:

    sudo firewall-cmd --list-all-zones
sudo firewall-cmd --zone=public --list-ports

    Ensure that the NodePorts exposed by your Ingress Controller (if using NodePort service type) are open, typically ports 30000-32767. Also, ensure that internal traffic for the Kubernetes CNI network (e.g., Flannel, Calico) is not blocked between nodes. Calico, for example, might require specific ports for BGP or IPIP tunnels.

    If you need to open a port (e.g., for a NodePort service):

    sudo firewall-cmd --zone=public --add-port=3xxxx/tcp --permanent
sudo firewall-cmd --reload

    If firewalld is overly restrictive, temporarily disabling it on a test node (NOT production) for diagnostic purposes might help isolate the issue, but remember to re-enable it.

    sudo systemctl stop firewalld
sudo systemctl disable firewalld # Temporary diagnostic only

By systematically working through these steps, you should be able to pinpoint the exact cause of your Kubernetes Ingress path routing 404 error on CentOS Stream or Rocky Linux and apply the appropriate fix.