This blog describes how to configure remote-write to send data from Prometheus running in your Azure Red Hat OpenShift (ARO) cluster to Azure managed Prometheus using Azure Active Directory authentication. Then query and view the data via Azure managed Prometheus and Grafana.
If you are wondering why send metrics to Azure managed Prometheus since ARO comes with pre-installed and configured Prometheus already? Here are couple of use cases. You may choose to use the managed service to centralize data from self-managed Prometheus clusters for long term data retention or to create a centralized view across your multiple clusters. If you don't have either of these use cases, then you don't need this remote-write.
Currently ARO doesn't support Managed Identity, hence we are using Azure Active Directory Service Principal approach. Managed Identity support for ARO is coming soon. One of the major drawbacks with Service Principal approach is that the client secret will have an expiry date.
Huge thanks to Vishwanath Narasimhan & Rakshith Padmanabha from Microsoft and Paul Czarkowski from Red Hat for providing guidance to accomplish this.
This article applies to the following cluster configurations:
Azure Red Hat OpenShift cluster
Azure Red Hat OpenShift cluster up and running. Prometheus comes pre-installed and configured for Azure Red Hat OpenShift 4.x clusters.
High level steps
Create an Azure Active Directory application
Create an Azure Monitor Workspace (i.e. Azure managed Prometheus, data from Azure managed Prometheus is stored in Azure Monitor Workspace)
Create an Azure managed Grafana and connect the Azure Monitor workspace to the Grafana
Assign Monitoring Metrics Publisher role on the data collection rule to the AAD App (i.e. to the Service Principal)
# Get Tenant ID
TENANT_ID=$(az account get-access-token --query tenant --output tsv)
# Create a Service Principal and get the Client Secret
SERVICE_PRINCIPAL_CLIENT_SECRET="$(az ad sp create-for-rbac --name umarm-$AROCLUSTER --query 'password' -otsv)"
# Create a Service Principal Client ID
SERVICE_PRINCIPAL_CLIENT_ID="$(az ad sp list --display-name umarm-$AROCLUSTER --query '.appId' -otsv)"
Create an Azure Monitor Workspace
Follow the procedure at Create an Azure Monitor Workspace to create Azure Monitor Workspace. This is the way of setting up Azure managed Prometheus and data from Azure managed Prometheus is stored in Azure Monitor Workspace.
Create an Azure Managed Grafana and connect the Azure Monitor workspace to the Grafana
To view your clusters consolidated CPU, Memory, etc. usage go to Dashboards in Grafana and select Azure Managed Prometheus. Then under Compute Resources section, select Kubernetes / Compute Resources / Cluster.
To view one of your Pod's memory or CPU usage, go to Dashboards in Grafana and select Azure Managed Prometheus. Then under Compute Resources section, select Kubernetes / Compute Resources / Nodes (Pods).
Select the necessary nodes from node dropdown and enter your Pod name under the Filters. Now you can view CPU and Memory Usage using Grafana dashboard for your Pod.