Azure Kubernetes Service Security Deep Dive – Part 5 (Securing Egress and Ingress)

Published Jan 25 2022 05:56 AM 2,055 Views
Microsoft

In the previous part of this series (part 4), we saw how you can secure ingress and egress traffic between pods within AKS cluster using Network Policy. But AKS is not an isolated service. Most of the time it’s part of bigger network that consists of other Azure services and computes. It’s evident that ingress and egress connections from those services are also essential. For example, your APIs running as stateless microservices within AKS need to connect Azure SQL DB or say Cosmos DB. In this article, we will discuss how to ensure security in such egress and ingress requirements. Fortunately, we have well documented resources within Azure documentation on this subject and hence we will only point out the important areas within those documentation.
Let’s talk about egress first. By default, AKS cluster has unbound external/egress access. Here are some notes and related sections from Azure Documentation that showcase process to secure or restrict egress traffic.

Note: 1
Network Security Group (NSG) at the subnet level is a standard process to restrict traffic to and from the subnet. But AKS outbound dependencies works on FQDNs and don't have assigned static addresses. Hence, NSG is not an option here.

Note: 2
AKS needs certain network, FQDN/application rules. Here are the detailed required network rules, FQDN: https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#required-outbound-network-rules-and-....  And optional FQDN: https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#optional-recommended-fqdn--applicati....

Note: 3
User Defined Route (UDR) along with Firewall is the most popular and standard way to restrict egress traffic from AKS. The documentation are here: https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#optional-recommended-fqdn--applicati... and https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#restrict-egress-traffic-using-azure-....  Here is a specific note from the documentation – “We recommend having a minimum of 20 Frontend IPs on the Azure Firewall for production scenarios to avoid incurring in SNAT port exhaustion issues”.

Note: 4
Last but not the least, to avoid limitations on the number of outbound flows of egress traffic through an Azure Load Balancer, you can use Managed NAT Gateway that allows up to 64,000 outbound UDP and TCP traffic flows per IP address with a maximum of 16 IP addresses. This feature is in preview at the time of writing this article. See here for detail: https://docs.microsoft.com/en-us/azure/aks/nat-gateway


Now, it’s time to talk about ingress. Ingress and ingress controllers are popular in Kubernetes world as they provide flexibility of standard level 7 load balancer including reverse proxy, configurable traffic routing, and TLS termination.

Note: 5
It’s well documented how to implement popular ingress controller like NGINX on AKS.  Here are the relevant links: https://docs.microsoft.com/en-us/azure/aks/ingress-basic,  https://docs.microsoft.com/en-us/azure/aks/http-application-routing.  Secure your NGINX ingress controller with SSL: https://docs.microsoft.com/en-us/azure/aks/ingress-own-tls,  https://docs.microsoft.com/en-us/azure/aks/ingress-tls?tabs=azure-cli,  https://docs.microsoft.com/en-us/azure/aks/ingress-static-ip.

 

Note: 6
Azure Application Gateway can also act as an Ingress Controller (AGIC) and it’s becoming popular as App Gateway provide advanced features like built-in Web Application Firewall (WAF), DDoS protection etc. https://docs.microsoft.com/en-us/azure/application-gateway/tutorial-ingress-controller-add-on-existi....


NGINX ingress controller is very efficient in some features like URL rewrite. Hence, it’s not rare to find an AKS implementation using both AGIC and NGINX to get the best of both the services. That’s pretty much it. Hope this article and the entire series is useful to you.

 

Other parts of these series:  Part1 | Part2 | Part3Part4 

 

 

Co-Authors
Version history
Last update:
‎Jan 07 2022 09:22 AM
Updated by: