Troubleshooting Kubernetes Ingress Controller 404 Errors on Path Routing (Ubuntu 20.04 LTS)

Resolve Kubernetes Ingress path routing 404 errors on Ubuntu 20.04 LTS. This guide diagnoses common misconfigurations and provides step-by-step fixes for Nginx Ingress Controller.

Resolve Kubernetes Ingress path routing 404 errors on Ubuntu 20.04 LTS. This guide diagnoses common misconfigurations and provides step-by-step fixes for Nginx Ingress Controller.

When deploying applications to Kubernetes and exposing them via an Ingress resource, encountering a "404 Not Found" error specifically when using path-based routing is a frustrating yet common issue. This typically means the Nginx Ingress Controller successfully received your request but couldn't find a matching rule to route it to your backend service, or it found a rule but couldn't reach the backend as expected. This guide will walk you through diagnosing and resolving these path routing issues on an Ubuntu 20.04 LTS environment.

Symptom & Error Signature

Users attempting to access your application via a configured Ingress path will receive an HTTP 404 Not Found response. This can manifest in several ways:

  1. Browser Output: A generic "404 Not Found" page, often rendered by the Ingress Controller's default backend if configured, or a blank page.
  2. curl Output:
    curl -I http://yourdomain.com/api/v1/users

    HTTP/1.1 404 Not Found
Server: nginx/1.21.6
Date: Thu, 25 May 2023 10:30:00 GMT
Content-Type: text/html
Content-Length: 153
Connection: keep-alive
  3. Nginx Ingress Controller Logs: When inspecting the logs of the Nginx Ingress Controller pod, you might see entries indicating no matching rules or upstream connection issues.
    # Example 1: No matching ingress rule
2023/05/25 10:30:00 [error] 36#36: *12345 no matching ingress rule for "yourdomain.com/api/v1/users"

# Example 2: Upstream connection refused (if rule was found but service unreachable)
2023/05/25 10:30:00 [error] 36#36: *12346 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.1, server: yourdomain.com, request: "GET /api/v1/users HTTP/1.1", upstream: "http://10.42.0.5:8080/api/v1/users", host: "yourdomain.com"
  4. kubectl describe ingress Output: While kubectl get ingress might show a healthy ingress resource, describe can sometimes reveal events related to rule parsing or backend issues.

Root Cause Analysis

A 404 error from the Ingress Controller on path routing typically stems from one or more of these underlying issues:

  1. Incorrect pathType Configuration: The pathType (e.g., Exact, Prefix, ImplementationSpecific) in the Ingress rule does not match how your application expects the path, or how the Ingress Controller processes it by default.
  2. Missing or Misconfigured rewrite-target Annotation: When using pathType: Prefix, the Nginx Ingress Controller by default strips the matched prefix before forwarding the request to the backend service. If your application expects the full path, a rewrite-target annotation is necessary.
  3. Service/Endpoint Mismatch: The Kubernetes Service referenced by your Ingress rule does not exist, or it exists but has no healthy pods backing it (i.e., its Endpoints are empty).
  4. Application Path Mismatch: Your application running in the pod is configured to serve content on a different base path than what the Ingress is routing (e.g., Ingress routes /api to the service, but the app expects requests at /).
  5. Ingress Controller Configuration: Issues with the Nginx Ingress Controller itself, such as an incorrect --default-backend-service or the Ingress resource not being picked up due to an ingressClassName mismatch.
  6. Network Policies: Restrictive Network Policies might be preventing the Ingress Controller pods from communicating with your backend Service pods.
  7. Pod Health and Readiness: The application pods backing your service might not be running, might be crashing, or might not be reporting as "Ready" to Kubernetes, preventing them from receiving traffic.

Step-by-Step Resolution

Follow these steps to systematically diagnose and resolve the 404 error.

1. Verify Ingress Resource Status and Configuration

First, ensure your Ingress resource is correctly defined and being picked up by the Ingress Controller.

# Replace <namespace> with your application's namespace
# Replace <ingress-name> with the name of your Ingress resource
kubectl get ingress -n <namespace> <ingress-name> -o yaml

Review the output carefully, paying close attention to:

  • spec.rules: Ensure the host and path are correct.
  • spec.rules.http.paths[].path: The exact path you're trying to reach.
  • spec.rules.http.paths[].pathType: This is crucial.
  • spec.rules.http.paths[].backend.service.name: The name of your Kubernetes Service.
  • spec.rules.http.paths[].backend.service.port.number: The port the service exposes.
  • metadata.annotations: Look for nginx.ingress.kubernetes.io/rewrite-target or kubernetes.io/ingress.class (if using older versions or multiple controllers).
  • spec.ingressClassName: Ensure this matches the ingressClass your controller is configured for (e.g., nginx).
# Get a more detailed view, including events
kubectl describe ingress -n <namespace> <ingress-name>

Check the Events section for any warnings or errors related to the Ingress configuration.

2. Inspect Nginx Ingress Controller Logs

The Ingress Controller logs are your primary source for understanding how it processes incoming requests.

# Find your Ingress Controller pod(s)
kubectl get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx

# Replace <ingress-controller-pod-name> with the actual pod name
kubectl logs -n ingress-nginx <ingress-controller-pod-name> --tail=100

Look for lines containing the host and path you're trying to reach. Search for messages like "no matching ingress rule" or "upstream connect error."

If you see "no matching ingress rule", it means the Ingress Controller couldn't find any Ingress object that matches the requested host and path. This often points to issues with pathType, the path regex, or the host definition.

3. Validate Service and Endpoint Health

If the Ingress Controller does find a rule but reports an upstream connection error, the problem likely lies with the backend Service or its associated pods.

# Replace <service-name> with the name from your Ingress backend
kubectl get service -n <namespace> <service-name>
kubectl describe service -n <namespace> <service-name>

Crucially, examine the Endpoints field in the describe service output.

  • If Endpoints is <none>, your Service is not routing traffic to any pods. This is a critical issue.
  • If Endpoints lists IP addresses, these are the IPs of your healthy application pods.

Now check the Endpoints object directly:

kubectl get endpoints -n <namespace> <service-name>

If the ENDPOINTS column is empty, proceed to debug your application pods (Step 4).

4. Debug Application Pods

If the Service has no healthy Endpoints, your application pods are the problem.

# Find your application pods (adjust selector as needed, e.g., app=my-app)
kubectl get pods -n <namespace> -l <your-app-selector-key>=<your-app-selector-value>

# Replace <pod-name> with one of your application pod names
kubectl describe pod -n <namespace> <pod-name>
kubectl logs -n <namespace> <pod-name> --tail=100
  • kubectl describe pod: Check the Events section for errors during pod creation, image pull, or container startup. Look at Readiness Gates and Liveness Probes status.
  • kubectl logs: See if your application is starting successfully, listening on the correct port, and if it's logging any errors related to its own request handling or internal configuration.

A common mistake is the application listening on a different port than what the Service (and thus Ingress) expects, or failing to start due to environmental issues.

Test direct connectivity to a pod: You can bypass the Service and Ingress to test if the application pod itself responds correctly to requests.

# Port-forward to your application pod
# This will listen on your local machine's port 8080 and forward to the pod's port 80
kubectl port-forward -n <namespace> <pod-name> 8080:<your-app-container-port>

In a new terminal, try to access your application directly via localhost:

curl http://localhost:8080/your-application-expected-path

If this curl also returns a 404 or fails, the problem is within your application or its configuration, not the Ingress or Service.

5. Review Ingress pathType and rewrite-target Annotations

This is one of the most frequent causes of 404 errors with path routing.

  • pathType: Exact:

    • path: /api/v1/users will only match /api/v1/users.
    • It will not match /api/v1/users/ or /api/v1/users/123.
    • Useful for specific API endpoints or static files.

  • pathType: Prefix:

    • path: /api will match /api, /api/, /api/v1/, /api/v1/users.
    • Crucially, by default, the matched prefix (/api in this case) is stripped before the request is forwarded to your backend service.
    • So, a request to yourdomain.com/api/v1/users will be forwarded to your service as /v1/users. If your application expects /api/v1/users, it will return a 404.

  • nginx.ingress.kubernetes.io/rewrite-target Annotation: This annotation allows you to rewrite the URL path before it's sent to the backend service.

    Scenario 1: Application expects root (/) but Ingress path is specific. Your application serves everything from its root path (/). You want yourdomain.com/myapp to route to / on your service.

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: / # Rewrites /myapp/* to /
spec:
  ingressClassName: nginx
  rules:
  - host: yourdomain.com
    http:
      paths:
      - path: /myapp
        pathType: Prefix
        backend:
          service:
            name: myapp-service
            port:
              number: 80

    Request yourdomain.com/myapp/foo will be rewritten to /foo for myapp-service.

    Scenario 2: Application expects the full path, even with Prefix matching. Your application expects yourdomain.com/api/v1/users to arrive as /api/v1/users at the service, but your Ingress path is /api with pathType: Prefix.

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2 # This uses regex capture groups
spec:
  ingressClassName: nginx
  rules:
  - host: yourdomain.com
    http:
      paths:
      - path: /api(/|$)(.*) # Regex to capture everything after /api
        pathType: ImplementationSpecific # Required for regex paths in Nginx Ingress Controller
        backend:
          service:
            name: api-service
            port:
              number: 8080

    Request yourdomain.com/api/v1/users will match /api(/|$)(.*), $1 will be / or empty, $2 will be v1/users. The rewrite target $2 ensures v1/users is sent to the backend. The leading / for the target is important.

    When using rewrite-target with regex, you must set pathType: ImplementationSpecific. This signals to the Ingress Controller that the path field contains a regex rather than a literal path.

    Simplified approach for preserving path (Nginx Ingress v1.2+): For Nginx Ingress Controller versions 1.2.0 and above, if you want pathType: Prefix to not strip the prefix and effectively behave like a passthrough, you can simply rewrite to the matched path:

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress-v2
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$request_uri # Uses Nginx variable to preserve original URI
spec:
  ingressClassName: nginx
  rules:
  - host: yourdomain.com
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 8080

    This is generally a cleaner approach than complex regexes if your only goal is to preserve the path.

6. Verify Ingress Class

If you have multiple Ingress Controllers deployed in your cluster or are using a non-default one, ensure your Ingress resource specifies the correct ingressClassName.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  # For older Kubernetes versions (pre-1.18) or specific configurations:
  # annotations:
  #   kubernetes.io/ingress.class: nginx 
spec:
  ingressClassName: nginx # Ensure this matches your Nginx Ingress Controller's class
  rules:
  # ...

If your Ingress Controller isn't configured with the nginx class, or you have a custom class name, this mismatch will prevent the controller from processing your Ingress resource. Check your Ingress Controller deployment arguments for --ingress-class.

7. Check Network Policies

If you are using Kubernetes Network Policies, ensure that they permit traffic from your Ingress Controller pods to your application Service pods.

# Get all network policies in your namespace
kubectl get networkpolicy -n <namespace>

# Describe a specific network policy if you suspect it's blocking traffic
kubectl describe networkpolicy -n <namespace> <policy-name>

The Ingress Controller pods (usually in ingress-nginx namespace) need to be able to initiate connections to your backend pods' IP addresses on the target containerPort.

By systematically working through these steps, you should be able to pinpoint the exact cause of your Kubernetes Ingress Controller path routing 404 errors and restore access to your applications.