Istio Service Mesh Observability in AKS

Introduction 

A service mesh is a dedicated infrastructure layer that manages service-to-service communication in microservices architectures. It is essential for managing communication between microservices in a distributed system, providing built-in security, traffic control, and observability.   

Istio is a powerful, open-source service mesh that simplifies managing, securing, and observing microservices communication. It joined Cloud Native Computing Foundation (CNCF) in 2022 and has become an industry standard for Service mesh operation.  

Azure Kubernetes Service (AKS) is a managed Kubernetes service provided by Microsoft Azure. It allows you to deploy, manage, and scale containerized applications using Kubernetes, without needing extensive container orchestration expertise. 

Observability in Istio Service Mesh is crucial for ensuring reliability, performance and security of microservices-based applications. Istio is a powerhouse when it comes to exposing telemetry and understanding the complex flow of traffic between applications. 

This article is a step-by-step guide for enabling Istio service mesh in AKS using Istio addon and enabling observability using managed Prometheus and Grafana. At the end we will discuss Advanced Container Networking Services (ACNS) addon in AKS, which enables Hubble to help visualize traffic flow within an AKS cluster / service mesh. I wanted to document the process as there are not enough articles available currently to achieve this in AKS and specifically none that talk about enabling Istio metrics export with mTLS enabled in AKS cluster (at the time of writing this article 😊). 

Metrics scraping architecture 

AKS Istio metrics scraping architecture

 Above is a simplified architecture diagram on how the metrics will get scraped in AKS by Prometheus. Prometheus is embedded into the azure monitor pods (ama-pods), and they will be doing the scraping based on the scraping configuration set. Each application pod will have a Istio-proxy container sidecar to control traffic and collect metrics, this also depends on which namespaces have sidecar auto-injection enabled or which pods are explicitly injected with sidecar. Hubble pods will also be running on the cluster utilizing the eBPF technology to scrape network flows using Layer 3. Prometheus will collect all these metrics and send it out to azure monitor workspace (customized Prometheus database) via private endpoint. Managed Grafana instance will then pull this data from azure monitor workspace over private endpoint again. 

Steps for configuring managed Prometheus, Grafana and Hubble 

1. Start with logging into AZ CLI with your account and selecting the default subscription and define some variables that you will use for creation of resource group and AKS cluster. 

# Define variables 
export MY_RESOURCE_GROUP_NAME="<your resource group name>"
export REGION="<region where you would like to deploy the cluster>" 
export MY_AKS_CLUSTER_NAME="<AKS cluster name>"

# Create a resource group 
az group create --name $MY_RESOURCE_GROUP_NAME --location $REGION 

# Create an AKS cluster 
az aks create --resource-group $MY_RESOURCE_GROUP_NAME --name $MY_AKS_CLUSTER_NAME --node-count 3 --generate-ssh-keys

 Once completed you should be able to see your AKS cluster in the Azure portal.  

 

 2. Get credentials for the AKS cluster and verify your connection.  

# Get the credentials for the AKS cluster 
az aks get-credentials --resource-group $MY_RESOURCE_GROUP_NAME --name $MY_AKS_CLUSTER_NAME 

# Verify the connection to your cluster 
kubectl get nodes

 If the connection is successful and the AKS cluster was created successfully you should see the nodes created as part of your aks cluster.  

  

3. Enable Istio addon for AKS (You might need to install aks-preview plugin for AZ CLI if not already installed). Then verify the installation of istio and enable sidecar injection in desired namespace  

# Enable istio addon on AKS cluster 
az aks mesh enable --resource-group $MY_RESOURCE_GROUP_NAME --name $MY_AKS_CLUSTER_NAME 

# Verify istiod (Istio control plane) pods are running successfully 
kubectl get pods -n aks-istio-system 

# Enable sidecar injection 
az aks show --resource-group $MY_RESOURCE_GROUP_NAME --name $MY_AKS_CLUSTER_NAME  --query 'serviceMeshProfile.istio.revisions'

 Based on the output of the above command use the appropriate label to enable sidecar injection, below “default” is the namespace where I am enabling sidecar injection 

kubectl label namespace default istio.io/rev=asm-1-22

 Sample output: 

 

4. Deploy sample application and verify its deployment  

# Deploy sample application 
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.18/samples/bookinfo/platform/kube/bookinfo.yaml 

# Verify services and pods 
kubectl get services 
kubectl get pods 
kubectl port-forward svc/productpage 12002:9080

 Sample Output: 

Above you will notice in output of “kubectl get pods” that each of the pods have 2 containers under READY column. This is because you had enabled sidecar injection in default namespace, the 2nd container in each pod is the istio-proxy container

After port forwarding your app to local port 12002, you should be able to access it: http://localhost:12002  

 

 

 5. Enable mTLS in your service mesh. This is one of the most important use cases of istio that it enables you to enforce mTLS so that only mTLS traffic is allowed in your mesh, improving your cluster security significantly.  

# Enable mTLS enforcement for default namespace in the cluster (copy / paste and run the entire code block till the 2nd EOF in terminal)
kubectl apply -n default -f - <<EOF
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:   
  name: default 
spec:   
  mtls:     
    mode: STRICT
EOF 

# Verify your policy got deployed 
kubectl get peerauthentication -n default

 Sample output: 

 

6. Now we will deploy managed prometheus and grafana. We will then link them with the AKS cluster. This will enable us to visualize prometheus based metrics from kubernetes on Grafana dashboard.  

# Create azure monitor resource (managed prometheus resource) 
export AZURE_MONITOR_NAME="<your desired name for managed prometheus resource>" 
az resource create --resource-group $MY_RESOURCE_GROUP_NAME --namespace microsoft.monitor --resource-type accounts --name $AZURE_MONITOR_NAME --location $REGION --properties '{}' 

# Create Azure Managed Grafana instance 
export GRAFANA_NAME="<your desired name for managed grafana resource>" 
az grafana create --name $GRAFANA_NAME --resource-group $MY_RESOURCE_GROUP_NAME --location $REGION 

# Link Azure Monitor and Azure Managed Grafana to the AKS cluster 
grafanaId=$(az grafana show --name $GRAFANA_NAME --resource-group $MY_RESOURCE_GROUP_NAME --query id --output tsv) 
azuremonitorId=$(az resource show --resource-group $MY_RESOURCE_GROUP_NAME --name $AZURE_MONITOR_NAME --resource-type "Microsoft.Monitor/accounts" --query id --output tsv) 
az aks update --name $MY_AKS_CLUSTER_NAME --resource-group $MY_RESOURCE_GROUP_NAME --enable-azure-monitor-metrics --azure-monitor-workspace-resource-id $azuremonitorId --grafana-resource-id $grafanaId 

# Verify Azure monitor pods are running 
kubectl get pods -o wide -n kube-system | grep ama-

 Sample output: 

 On Azure portal, you can check that the new resources are created:  

 You should then open the grafana instance and click on the instance URL to open your managed grafana instance. If you are not able to do so, assign yourself Grafana Admin role under Access control pane in Grafana resource on Azure:  

 

 

 7. Now you will need to configure a job and configmap for prometheus to scrape metrics from istio. Download the configmap prometheus-config from here. 

# Create job and configmap for scraping istio metrics with prometheus 
kubectl create configmap ama-metrics-prometheus-config --from-file=prometheus-config -n kube-system

 Wait for about 10-15 mins and then verify whether istio metrics are getting scraped from your cluster. Go to prometheus resource on Azure -> Metrics on the left pane -> Select “istio_requests_total” and run query. You should be able to see data popping up after that.  

 

8. Import Istio Grafana dashboards to your managed Grafana instance. For doing this first find out the version of istio you are running on your cluster 

# Get Istio version Installed for importing specific dashboards 
az aks show --resource-group $MY_RESOURCE_GROUP_NAME --name $MY_AKS_CLUSTER_NAME  --query 'serviceMeshProfile.istio.revisions'

 Sample output: 

 After this go to the following dashboards and download the specific version based on your istio version: 

• Istio Mesh Dashboard | Grafana Labs 

• Istio Control Plane Dashboard | Grafana Labs 

• Istio Service SLO Demo | Grafana Labs (Only 1 version is available here) 

For each of the dashboards downloaded above, click on dashboards on Grafana and New->Import option on top right corner. After clicking on import upload the downloaded json file of the dashboard and click on import. Remember to select Azure managed prometheus as data source before importing. 

 

 Post this you should be able to see istio metrics displayed on the Grafana dashboards: 

 

 

  

9. Now that you have exported Istio metrics and created dashboards, we will now need to see how to visualize traffic flow graphs in AKS. This is critical as with complex service mesh, you will need to understand how your traffic is flowing. The standard way to do this is either using Kiali or Jaeger, which currently are not supported with Istio addon for AKS. We will use Hubble, which is an eBPF technology developed by Cilium to scrape network flows using Layer 3 (so it would be more efficient). Hubble is ported to non-cilium AKS clusters using retina which is available using the Advanced Container Networking Service (ACNS) addon. You can download hubble-ui.yaml from here. 

# Enable ACNS for the AKS cluster 
az aks update --resource-group $MY_RESOURCE_GROUP_NAME --name $MY_AKS_CLUSTER_NAME --enable-acns 

# Setup Hubble UI 
kubectl apply -f hubble-ui.yaml kubectl -n kube-system port-forward svc/hubble-ui 12000:80

Sample output: 

Navigate to http://localhost:12000 on your browser to open Hubble UI 

 

Conclusion 

We have learned how to configure observability for Istio metrics using managed Prometheus and Grafana on AKS and visualize network flows using Hubble. You can find the commands and yaml files used in this article here. Let me know if you face any issues during this implementation via comments. Thank you for reading this article! Happy learning!