Configure remote write to send data from ARO to Azure managed Prometheus using AAD authentication.
Published Sep 21 2023 04:48 PM 2,994 Views

This blog describes how to configure remote-write to send data from Prometheus running in your Azure Red Hat OpenShift (ARO) cluster to Azure managed Prometheus using Azure Active Directory authentication. Then query and view the data via Azure managed Prometheus and Grafana. 

 

If you are wondering why send metrics to Azure managed Prometheus since ARO comes with pre-installed and configured Prometheus already? Here are couple of use cases. You may choose to use the managed service to centralize data from self-managed Prometheus clusters for long term data retention or to create a centralized view across your multiple clusters. If you don't have either of these use cases, then you don't need this remote-write.

 

Currently ARO doesn't support Managed Identity, hence we are using Azure Active Directory Service Principal approach. Managed Identity support for ARO is coming soon. One of the major drawbacks with Service Principal approach is that the client secret will have an expiry date.

 

Credit

Huge thanks to Vishwanath Narasimhan & Rakshith Padmanabha from Microsoft and Paul Czarkowski from Red Hat for providing guidance to accomplish this.

 

Cluster configurations

This article applies to the following cluster configurations:

  • Azure Red Hat OpenShift cluster

Prerequisites

  • Azure Red Hat OpenShift cluster up and running. Prometheus comes pre-installed and configured for Azure Red Hat OpenShift 4.x clusters.

High level steps

  • Create an Azure Active Directory application
  • Create an Azure Monitor Workspace (i.e. Azure managed Prometheus, data from Azure managed Prometheus is stored in Azure Monitor Workspace)
  • Create an Azure managed Grafana and connect the Azure Monitor workspace to the Grafana
  • Assign Monitoring Metrics Publisher role on the data collection rule to the AAD App (i.e. to the Service Principal)
  • Create Secret object in ARO
  • Update ARO's cluster monitoring config
  • View the metric data in Prometheus and Grafana

 

Create Azure Active Directory application

Follow the procedure at Register an application with Azure AD and create a service principal to register an application for Prometheus remote-write and create a service principal. Also get the Tenant ID.

 

 

 

# Get Tenant ID
TENANT_ID=$(az account get-access-token --query tenant --output tsv)

# Create a Service Principal and get the Client Secret
SERVICE_PRINCIPAL_CLIENT_SECRET="$(az ad sp create-for-rbac --name umarm-$AROCLUSTER --query 'password' -otsv)"

# Create a Service Principal Client ID
SERVICE_PRINCIPAL_CLIENT_ID="$(az ad sp list --display-name umarm-$AROCLUSTER --query '[0].appId' -otsv)"

 

 

 

Create an Azure Monitor Workspace

Follow the procedure at Create an Azure Monitor Workspace to create Azure Monitor Workspace. This is the way of setting up Azure managed Prometheus and data from Azure managed Prometheus is stored in Azure Monitor Workspace.

 

Create an Azure Managed Grafana and connect the Azure Monitor workspace to the Grafana

Follow the procedure at Create an Azure Managed Grafana to create Azure Managed Grafana instance.

Next follow the procedure at Link a Grafana workspace to connect the Azure Monitor workspace to the Azure managed Grafana instance.

 

Assign Monitoring Metrics Publisher role the AAD App (i.e. to the Service Principal)

Follow the procedure at Assign Monitoring Metrics Publisher role to the Service Principal to assign  Monitoring Metrics Publisher role on the data collection rule associated with your Azure Monitor workspace to the Service Principal.

 

Create Secret object in ARO

Create a secret object in ARO using the Service Principal client ID and client secret values from above.

 

 

cat << EOF | oc apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: oauth2-credentials
stringData:
  id: "${SERVICE_PRINCIPAL_CLIENT_ID}"
  secret: "${SERVICE_PRINCIPAL_CLIENT_SECRET}"
EOF

 

 

 

Update ARO's cluster monitoring config

Update ARO's cluster monitoring config to remote write on the Azure managed Prometheus.

To edit cluster-monitoring-config file run the following command.

 

 

 

oc edit -n openshift-monitoring cm cluster-monitoring-config

 

 

 

Edit this config file in editor like Vim with following configuration.

Make sure to replace {{INGESTION-URL}} value below with Metrics ingestion endpoint from the Overview page for the Azure Monitor workspace.

 

 

 

data:
  config.yaml: |
    prometheusK8s:
      remoteWrite:
        - url: "{{INGESTION-URL}}"
          oauth2:
            clientId:
              secret:
                name: oauth2-credentials
                key: id
            clientSecret:
              name: oauth2-credentials
              key: secret
            tokenUrl: "https://login.microsoftonline.com/$TENANT_ID/oauth2/v2.0/token"
            scopes:
              - "https://monitor.azure.com/.default"

 

 

 

View the metric data in Prometheus and Grafana

You can either view your clusters CPU/Memory usage or assuming you are running an application in ARO already, you can start viewing your application resources metrics in Prometheus and Grafana. 

 

Querying metrics in Prometheus

Go to Prometheus explorer in Azure Monitor Workspace and run the following PromQL queries.

 

  • The following query retrieves the current memory usage in bytes for the specified pod's container.

container_memory_usage_bytes{pod="<pod_name>", namespace="<namespace>", container="<container_name>"}

 

For example:

container_memory_usage_bytes{pod="mypod-azure-private-file", namespace="default", container="mypod-azure-private-file-container"}

 

UmarMohamedUsman_0-1695326505874.png

 

  • This query retrieves the total CPU usage in seconds for the specified pod's container.

container_cpu_usage_seconds_total{pod="<pod_name>", namespace="<namespace>", container="<container_name>"}

 

For example:

container_cpu_usage_seconds_total{pod="mypod-azure-private-file", namespace="default", container="mypod-azure-private-file-container"}

 

UmarMohamedUsman_1-1695326545183.png

 

Viewing metrics in Grafana

To view your clusters consolidated CPU, Memory, etc. usage go to Dashboards in Grafana and select Azure Managed Prometheus. Then under Compute Resources section, select Kubernetes / Compute Resources / Cluster.

UmarMohamedUsman_0-1695339789160.pngUmarMohamedUsman_1-1695339897544.png

   

To view one of your Pod's memory or CPU usage, go to Dashboards in Grafana and select Azure Managed Prometheus. Then under Compute Resources section, select Kubernetes / Compute Resources / Nodes (Pods).

 

UmarMohamedUsman_2-1695326584680.png

 

Select the necessary nodes from node dropdown and enter your Pod name under the Filters. Now you can view CPU and Memory Usage using Grafana dashboard for your Pod.

 

UmarMohamedUsman_3-1695326584682.png

 

 

 

Co-Authors
Version history
Last update:
‎Oct 06 2023 03:04 PM
Updated by: