Kubernetes Nginx Ingress Controller Logs
Published Nov 25 2022 06:52 AM 857 Views
Microsoft

By default, the Kubernetes Nginx Ingress Controller writes logs and traces to stdout / stderr.

ormeikopmsft_0-1669387806612.png

These logs and traces are stored in the ContainerLog table. You can navigate to an autogenerated query from inside the Container Insights blade from the Azure Portal.

ormeikopmsft_1-1669387806625.png

ormeikopmsft_2-1669387806631.png

During an engagement, we faced an issue with viewing these logs and traces. We could view the logs in the “Live Logs” tab inside the Portal, but none of these got sent to Log Analytics workspace and we were unable to retain and run Kusto queries against that data.

ormeikopmsft_3-1669387806635.png

As it turned out, after a troubleshooting session with a colleague, the reason the Nginx Ingress Controller logs and traces were not sent in Log Analytics workspace was the fact that the controller had been deployed to the kube-system namespace, which happens to be excluded by default from the container log collection, based on the below note.

ormeikopmsft_4-1669387806636.png

To solve this, the recommended approach, based also on the example in the official Microsoft Learn documentation, is to redeploy the Ingress Controller to a different namespace.

Taking it to the next level, we wanted to do this with zero downtime. For that, a secondary Ingress Controller needs to be created. Then, the ingress routes need to be duplicated to use the secondary ingress controller. Finally, you should do a swap in the Firewall Appliance to redirect traffic to the new Ingress IP address.

Below you can find the detailed steps to deploy and configure a secondary ingress controller:

  • Create an additional ingress controller on a different / appropriate namespace, providing a custom ingressClass (e.g., “nginx-v2”), given the default “nginx” ingressClass will already be in use by the existing ingress controller. The helm install command is simplified for readability, you will probably have additional parameters as per your script, but the highlighted ones are required:

helm install nginx-ingress ingress-nginx/ingress-nginx \

    --namespace ingress-nginx-v2 \

    --create-namespace \

    --set controller.ingressClass=nginx-v2 \

    --set controller.ingressClassResource.name=nginx-v2 \

    --set controller.ingressClassResource.controllerValue="k8s.io/nginx-v2"

  • At this point, there should be two ingress controllers on different namespaces and two different ingressClass definitions:

kubectl get svc --selector=app.kubernetes.io/name=ingress-nginx -A

 

NAMESPACE          NAME                                               TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                      AGE

ingress-nginx-v2   nginx-ingress-ingress-nginx-controller             LoadBalancer   11.0.247.238   51.1.1.2   80:32746/TCP,443:31787/TCP   20m

ingress-nginx-v2   nginx-ingress-ingress-nginx-controller-admission   ClusterIP      11.0.114.217   <none>           443/TCP                      20m

ingress-nginx      ingress-nginx-controller                           LoadBalancer   11.0.220.7     51.1.1.1   80:32040/TCP,443:30183/TCP   15d

ingress-nginx      ingress-nginx-controller-admission                 ClusterIP      11.0.161.45    <none>           443/TCP                      15d

 

 

kubectl get ingressclass

 

NAME       CONTROLLER             PARAMETERS   AGE

nginx      k8s.io/ingress-nginx   <none>       15d

nginx-v2   k8s.io/nginx-v2        <none>       23m

  • Duplicate the existing ingress routes, as they will be referencing the original ingress controller:

k get ingress -A

 

NAMESPACE   NAME              CLASS   HOSTS   ADDRESS          PORTS   AGE

contoso     contoso-ingress   nginx   *       51.138.206.238   80      15d

  • This means you need to create an exact copy of all ingress routes, changing the “ingressClassName” to the newly created ingressClass:

cat << EOF | kubectl apply -f -

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

  name: contoso-ingress-v2

  namespace: contoso

  annotations:

    nginx.ingress.kubernetes.io/ssl-redirect: "false"

    nginx.ingress.kubernetes.io/use-regex: "true"

    nginx.ingress.kubernetes.io/rewrite-target: /$2

spec:

  ingressClassName: nginx-v2

  rules:

  - http:

      paths:

      - path: /contoso(/|$)(.*)

        pathType: Prefix

        backend:

          service:

            name: contoso-webapp

            port:

              number: 80

EOF

 

ingress.networking.k8s.io/contoso-ingress-v2 created

 

 

kubectl get ingress -A

 

NAMESPACE   NAME                 CLASS      HOSTS   ADDRESS          PORTS   AGE

contoso     contoso-ingress      nginx      *       1.1.1.1          80      15d

contoso     contoso-ingress-v2   nginx-v2   *       1.1.1.2          80      85s

At this point, you can access the applications using both ingress controllers, and you should be able to redirect all traffic to the new ingress after performing the necessary validations.

Co-Authors
Version history
Last update:
‎Nov 25 2022 06:55 AM
Updated by: