Kubernetes Nginx Ingress Controller Path Routing 404 Error Troubleshooting on Ubuntu 22.04 LTS

Fix Kubernetes Nginx Ingress path-based routing 404 errors on Ubuntu 22.04 LTS. Debug IngressClass, rules, service backend, and controller logs for proper traffic flow.

Introduction

Encountering a 404 Not Found error when accessing an application through a Kubernetes Ingress Controller, specifically with path-based routing, is a common but frustrating issue for DevOps engineers. This typically indicates that while your request might be reaching the Nginx Ingress Controller, the controller itself cannot find a matching Ingress rule or a valid backend service for the requested path. This guide provides a highly technical, step-by-step approach to diagnose and resolve such 404 errors on systems running Ubuntu 22.04 LTS with a Nginx Ingress Controller.

Symptom & Error Signature

Users attempting to access an application via a specific URL path configured in a Kubernetes Ingress resource will receive a 404 Not Found error in their browser. This often happens even when the root path (/) or other paths on the same host might be working correctly, or if the service is directly accessible via NodePort or LoadBalancer.

Typical Browser Output: A standard 404 Not Found page, sometimes branded by the Nginx Ingress Controller's default backend if one is configured, or simply the application's own 404 page.

curl Output:

$ curl -v http://your-domain.com/your-app-path

*   Trying 203.0.113.10...
* Connected to your-domain.com (203.0.113.10) port 80 (#0)
> GET /your-app-path HTTP/1.1
> Host: your-domain.com
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Date: Thu, 03 Oct 2024 10:30:00 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 23
< Connection: keep-alive
< Server: nginx/1.23.4
< X-Served-By: default-http-backend
<
404 Not Found
* Connection #0 to host your-domain.com left intact

The X-Served-By: default-http-backend header (if present) is a strong indicator that the Nginx Ingress Controller received the request but couldn't find a matching rule for it, forwarding it to its default 404 handler.

Nginx Ingress Controller Logs: When inspecting the logs of the Nginx Ingress Controller pod, you'll typically find entries similar to these for requests resulting in a 404:

203.0.113.1 - [203.0.113.1] - - [03/Oct/2024:10:30:00 +0000] "GET /your-app-path HTTP/1.1" 404 23 "-" "curl/7.81.0" 0.000 "-" "-" -
2024/10/03 10:30:00 [error] 32#32: *12345 no matching ingress rule for "your-domain.com/your-app-path", client: 203.0.113.1, server: 0.0.0.0:80, request: "GET /your-app-path HTTP/1.1", host: "your-domain.com"

The "no matching ingress rule" message is the most direct indicator that the Ingress controller did not find an Ingress resource with a host and path combination that matches the incoming request.

Root Cause Analysis

A 404 error during Kubernetes Ingress path routing usually stems from one or more of these underlying issues:

1. Ingress Class Mismatch or Absence: The Ingress controller is configured to watch for specific IngressClass resources, but your Ingress object either specifies a different ingressClassName or omits it entirely, causing the controller to ignore it.
2. Incorrect Path Definition:
   • pathType Misconfiguration: The pathType (e.g., Prefix, Exact, ImplementationSpecific) in the Ingress rule does not correctly match the incoming request path. Prefix is often misunderstood, especially with trailing slashes.
   • Regex Errors: If pathType: ImplementationSpecific is used with Nginx's regular expression annotations, the regex might be incorrect or not matching.
   • Missing or Incorrect Path: The specified path in the Ingress resource simply doesn't match the actual URL path being requested.
3. Backend Service Issues:
   • Service Not Found: The service.name specified in the Ingress rule points to a non-existent, misspelled, or incorrectly namespaced Kubernetes Service.
   • Service Port Mismatch: The service.port.number (or service.port.name) in the Ingress rule does not match the actual port exposed by the target Service.
   • Service Unavailable: The backend Service itself has no healthy endpoints (e.g., all pods are crashing, in Pending state, or scaled to zero).
   • Application Path Mismatch: The application within the pod expects a different base path than what the Ingress is forwarding, and no rewrite rules are in place.
4. Nginx Ingress Controller Annotations:
   • rewrite-target Misconfiguration: The nginx.ingress.kubernetes.io/rewrite-target annotation is incorrectly set, causing the request path to be rewritten to a non-existent endpoint on the backend service.
   • use-regex Issues: The nginx.ingress.kubernetes.io/use-regex annotation, if used, might be combined with a faulty regular expression in the path.
5. External Load Balancer or DNS Configuration: While less common for path 404s, ensuring your external Load Balancer (if applicable) or DNS entries correctly point to the Nginx Ingress Controller's external IP or hostname is a fundamental prerequisite. A 404 from the Ingress controller means it received the request.
6. Firewall Rules: Less likely to cause a 404 from the Ingress controller, but ensure no network policies or external firewalls are blocking traffic from the Ingress controller to the backend services on the pod network.

Step-by-Step Resolution

Follow these steps meticulously to pinpoint and resolve the 404 error.

1. Verify Ingress Controller Status and Ingress Class

Ensure your Nginx Ingress Controller is running correctly and that your Ingress resource is being picked up by it.

1. Check Ingress Controller Pods:

   kubectl get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx
   

   Ensure all pods are in a Running state. If not, inspect their logs (kubectl logs <pod-name> -n ingress-nginx).

2. Verify IngressClass Definition: First, list your IngressClass resources:

   kubectl get ingressclass
   

   Look for an IngressClass that defines your Nginx controller (e.g., nginx, nginx-example).

3. Inspect Your Ingress Resource's ingressClassName:

   kubectl get ingress <your-ingress-name> -n <your-namespace> -o yaml
   

   Ensure the spec.ingressClassName field in your Ingress resource exactly matches the name of an existing IngressClass resource your controller is configured to watch. If ingressClassName is omitted, and you have a default IngressClass defined (ingressclass.kubernetes.io/is-default-class: "true" annotation), the controller might pick it up. If no default is set, an omitted ingressClassName might lead to the Ingress being ignored.

   A common mistake is having a mismatch or typo in ingressClassName. If the Ingress controller doesn't pick up the Ingress resource, it cannot route traffic for it.

4. Check Ingress Controller Logs for Parsing Errors: The Ingress controller logs will often show if an Ingress resource is being processed or if there are errors parsing its configuration.

   kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx | grep "Successfully added"
   kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx | grep "Error processing Ingress"
   

   Look for successful additions of your Ingress resource or any related errors.

2. Inspect Ingress Resource Configuration

The most frequent cause of path-based 404s is incorrect path or pathType configuration.

1. Retrieve and Examine Your Ingress Resource:

   kubectl get ingress <your-ingress-name> -n <your-namespace> -o yaml
   

   Pay close attention to the spec.rules section, specifically host, http.paths, path, pathType, backend.service.name, and backend.service.port.number.

   Example Ingress Configuration:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: my-app-ingress
     namespace: my-namespace
     annotations:
       # Potentially relevant annotations (see Step 5)
       # nginx.ingress.kubernetes.io/rewrite-target: /$2
       # nginx.ingress.kubernetes.io/use-regex: "true"
   spec:
     ingressClassName: nginx # Must match your IngressClass
     rules:
     - host: your-domain.com
       http:
         paths:
         - path: /my-app(/|$)(.*) # Example for regex, typically simpler like /my-app
           pathType: Prefix # Or Exact, or ImplementationSpecific
           backend:
             service:
               name: my-app-service
               port:
                 number: 80
   

2. Understand pathType:

   • Prefix: Matches URL path segments. For /foo, it matches /foo, /foo/, and /foo/bar. A path / matches all URLs. If multiple prefixes match, the longest matching path wins.
     • Important: A Prefix path /foo will match /foo and /foo/bar. If the backend expects / (root), you might need a rewrite (see Step 5).
   • Exact: Matches the URL path exactly and case-sensitively. /foo only matches /foo, not /foo/ or /foo/bar.
   • ImplementationSpecific: The matching behavior depends on the IngressClass. For Nginx, this often implies regex support, especially when combined with nginx.ingress.kubernetes.io/use-regex: "true".

   Misunderstanding pathType is a leading cause of 404s. If your application expects requests to its root (/) but your Ingress path is /my-app, a pathType: Prefix on /my-app without a rewrite-target means the request arrives at the backend as /my-app/something, which the application might not handle.

3. Validate Backend Service and Pods

Even if the Ingress rule matches, a faulty backend service will result in a 404 (or 503 if the service has no healthy endpoints).

1. Verify Service Existence and Configuration:

   kubectl get service <my-app-service> -n <my-namespace> -o yaml
   

   Ensure my-app-service exists, is in the correct namespace, and its spec.ports section correctly exposes the port (80 in our example) that your application is listening on. Check that the selector matches your application's pod labels.

2. Check Service Endpoints:

   kubectl get endpoints <my-app-service> -n <my-namespace>
   

   You should see at least one address listed under ENDPOINTS. If not, your service selector isn't matching any running pods, or the pods are unhealthy.

3. Inspect Application Pod Status:

   kubectl get pods -n <my-namespace> -l app=my-app-label
   

   Ensure pods are Running and Ready. If they are in CrashLoopBackOff, Pending, or Error states, inspect their logs:

   kubectl logs <my-app-pod-name> -n <my-namespace>
   kubectl describe pod <my-app-pod-name> -n <my-namespace>
   

4. Directly Test the Service (Bypass Ingress): This is a critical diagnostic step. Spin up a temporary pod in your cluster and try to curl your service directly using its cluster IP or DNS name.

   kubectl run -it --rm debug --image=busybox:latest -- /bin/sh
   # Inside the debug pod:
   wget -O- http://my-app-service.<my-namespace>.svc.cluster.local:80/
   # Or for a specific path:
   wget -O- http://my-app-service.<my-namespace>.svc.cluster.local:80/expected-app-path
   exit
   

   If this direct call also returns a 404 (or other errors), the problem lies with your application or service configuration, not the Ingress controller.

4. Analyze Nginx Ingress Controller Logs

The controller's logs are invaluable for understanding how it processes requests.

1. Tail Controller Logs:

   kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --tail=200 -f
   

   While tailing, try to access your problematic URL. Look for lines corresponding to your request. Specifically, search for messages like:

   • no matching ingress rule
   • serving default backend
   • rewriting request (if annotations are used)
   • Your request's HTTP status code (e.g., 404).

   Many Ingress controllers log a unique request_id or similar. If yours does, you can grep for that ID to trace a single request through the logs.

5. Check Nginx Ingress Annotations (Rewrite Rules)

If you're using path-based routing, especially with pathType: Prefix and your application expects a root path, nginx.ingress.kubernetes.io/rewrite-target is crucial.

1. Examine Annotations:

   kubectl get ingress <your-ingress-name> -n <your-namespace> -o yaml
   

   Look under metadata.annotations.

2. Common rewrite-target Scenario: If your Ingress path is /my-app and your backend application serves content from its root (/), you likely need a rewrite.

   • Incorrect (or missing) annotation: Request to your-domain.com/my-app/foo gets sent to backend as /my-app/foo. Backend returns 404 because it expects /foo.
   • Correct annotation:

     annotations:
       nginx.ingress.kubernetes.io/rewrite-target: /$1
       nginx.ingress.kubernetes.io/use-regex: "true" # Required for regex in path
     spec:
       rules:
       - host: your-domain.com
         http:
           paths:
           - path: /my-app/(.*) # Capture everything after /my-app
             pathType: ImplementationSpecific # Or Prefix, but ImplementationSpecific + regex is more precise
             backend:
               service:
                 name: my-app-service
                 port:
                   number: 80
     

     In this case, a request to your-domain.com/my-app/foo would be rewritten to /foo before being sent to my-app-service.

   Misconfigured rewrite-target annotations, especially with incorrect regex capture groups ($1, $2, etc.), are a very common cause of 404 errors. Ensure your regex in the path correctly captures the desired part of the URL, and your rewrite-target refers to the correct capture group.

6. Debug with Nginx Configuration in Controller

The Nginx Ingress Controller dynamically generates Nginx configuration files based on your Ingress resources. Inspecting these files directly can reveal exactly how Nginx is processing your paths.

1. Access the Controller Pod's Shell:

   kubectl exec -it <nginx-ingress-controller-pod> -n ingress-nginx -- bash
   

   (Replace <nginx-ingress-controller-pod> with the actual pod name from kubectl get pods -n ingress-nginx).

2. Inspect Generated Nginx Configuration: Inside the pod, the main configuration is usually at /etc/nginx/nginx.conf. Included files for specific Ingresses are often in directories like /etc/nginx/conf.d/.

   cat /etc/nginx/nginx.conf
   # Or, for specific virtual hosts/locations:
   cat /etc/nginx/conf.d/default.conf # This might be where your host is defined
   # Look for server blocks (your-domain.com) and location blocks (/your-app-path)
   

   Examine the server block for your host and the location block for your path. Verify that the proxy_pass directive points to the correct backend service and port, and that any rewrite rules (if annotations are used) are present and correctly formed.

   Example Snippet to Look For:

   server {
       listen 80;
       server_name your-domain.com;
   
       location ~* ^/my-app/(.*) { # Matches the path regex from Ingress
           # Possible rewrite rule if annotation was used
           rewrite ^/my-app/(.*) /$1 break;
           
           proxy_set_header Host $host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_pass http://upstream_balancer_my-app-service_80; # Points to the backend service
       }
       # ... other locations or default backend ...
   }
   

   If your path or rewrite rules are not present as expected in this configuration, it indicates an issue with how the Ingress controller is interpreting your Ingress resource (Step 2 and 5).

7. Firewall and Security Group Considerations

While less likely to cause a path-specific 404 (as traffic reaches the controller), ensure no network policies or cloud provider security groups are blocking internal cluster traffic.

• Kubernetes Network Policies: If you have Network Policies enabled, ensure they permit traffic from the Nginx Ingress Controller pods to your application's service pods on the relevant ports.

  kubectl get networkpolicy -n <your-namespace>
  

• Cloud Provider Security Groups: If your Kubernetes cluster is on a cloud provider (AWS, GCP, Azure), verify that the security group attached to your worker nodes (where the Ingress controller and application pods run) allows traffic on the necessary ports (e.g., 80/443 from Load Balancer, and internal pod network traffic).

By systematically working through these steps, you should be able to identify the precise misconfiguration causing the 404 error in your Kubernetes Ingress path routing setup. Remember to apply changes and re-test after each potential fix.