End to End TLS with Application Gateway + AGIC And Pods Service Discovery Using Azure Private DNS

[vkuma297]

HI Microsoft Team, 

Lately we have exploring the powerful addon , AGIC which fits in right way with AKS and App gateway. However, we are trying to figure out some problems we are facing. We are trying to achieve End to End 

TLS ( FLow : client -> app gateway -> backend pools ( AKS + AGIC enabled ) 

Also, we have this flow setup in 2 regions . We need to enable communications between applications deployed in AKS clusters in both the regions. For this purpose, we have created Private DNS Zone and added ExternalDNS add on ( which generally detects hostname and creates DNS A record in this private DNS Zone). We were able to integrate this ExternalDNS add on. 

Problem1 : When we deploy apps into AKS using hostnames, all the app dns records are pointing to App gateway public ip. 

Desired : We need to enable communication between apps deployed into AKS clusters in 2 regions talk with each region with out app gateway. Something like using private DNS with A record mapped to  unique public ip to this application exposed as Service type Loadbalancer. We need to accomplish this along with AGIC in place ( it should not point to app gateway public ip, instead point to public ip of app exposed as service type Loadbalancer)

Problem 2 : End to End TLS between client to app gateway : 

I believe there are some docs around the same, Does app gateway listener has capability to upload the cert and do TLS encryption between client and app gateway?

If yes, can we have any related documentation to follow ? 

Problem 2 : End to End TLS between app gateway to AKS : 

we have seen some articles on this, we have figured out 2 things out of this : 

a. We can upload certs to App gateway and with the help of AGIC annotation : AppGw SSL Certificate

it creates a http listener and updates app gateway. 

Does this solve end to end TLS between app gateway to AKS pods ?

b. we can go to AKS cluster, using azure CSI, we can reference or mount the certs stored in Azure key vault into AKS pods and enable TLS. 

Does this solve end to end TLS between app gateway to AKS pods ?

Or we need a + b to solve for End to End TLS between app gateway to AKS 

[#2]

@vkuma297 For the issues and concerns  you described you should consider using a service mesh product like LinkerD (https://linkerd.io/).  This has ingress controller functionality, policy for mTLS, and cross-cluster communication.

Alternatively look at the availability zones implementation of AKS https://docs.microsoft.com/en-us/azure/aks/availability-zones which is different than just creating a base AKS install.  This guarantees cross-cluster communication, but would still require extra configuration for mTLS.

By default with the MS scripts from some of the reference architectures, it's by design that all inbound communication go through the API Gateway, so some of your behaviour may be expected/by default.