Store Prometheus Metrics with Thanos, Azure Storage and Azure Kubernetes Service (AKS)

Thanks for the explanation.

I performed the same steps although did not start from prometheus.yml but rather used the kube-prometheus-community helm chart few changes. Below are the steps

• Create a storage account and then create a secret as mentioned in your steps 

type: AZURE
config: 
  storage_account: "xxxxx"
  storage_account_key: "xxxxxx"
  container: "thanos"
  max_retries: 0

• Create the thanos-objstore-config secret and update the prometheus.prometheusSpec.thanos field to below

thanos:
  objectStorageConfig:
    key: thanos.yaml
    name: thanos-objectstore-config
  image: quay.io/thanos/thanos:v0.13.0

• Install the kube-prometheus-community helm chart with few changes in values.yaml.
• Was getting below error 

RESPONSE Status: 403 This request is not authorized to perform this operation.\n   Content-Length: [246]\n   Content-Type: [application/xml]\n   Date: [Fri, 25 Feb 2022 16:15:27 GMT]\n   Server: [Microsoft-HTTPAPI/2.0]\n   X-Ms-Error-Code: [AuthorizationFailure]\n   X-Ms-Request-Id: [eb889395-601e-000c-2a62-2a52e7000000]\n\n\n\nerror creating Azure blob container:​

• So created a managed identity and then an AzureIdentity and AzureIdentitybinding with name thanos
• Then added aadpodidbinding: thanos label to prometheus statefulset to bind the pods for this sts with the identity created. Assuming that this would work, it still does not work. The prometheus pods are still failing with same error

There is also an issue opened recetly on github https://github.com/thanos-io/thanos/issues/5172 

Kindly help here how to proceed.