We are on a quest to remove credentials where we can in dealing with Azure. Part 1 created a connection between GitHub and Azure using "federated credentials":
In part 2 we got started with setting up Azure App Services with EasyAuth:
At the end of part 2 we were kind of unsatisfied with the fact that while we have no secrets anywhere in our GitHub repository it is possible to retrieve a client secret either by glancing through the deployment logs or in the Azure Portal. For reasons that should be understandable we don't really like this.
As a lead-in to this post I'll quote myself from part 1: The preferred option is to get rid of this altogether by using "managed identities", but if you cannot avoid having keys put them somewhere safe like Azure Key Vault.
These are the the mechanisms we want to use to solve the not-so-secret secret issue.
Microsoft has a decent intro to Managed Identities:
https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview
The main point of managed identities is avoiding credentials so it's right up our alley. However, adding an identity to our web app adds no value on its own if we store and generate the clientSecret the same way as before. We also need to store our secrets somewhere safe; for most purposes in Azure this will be through using Azure Key Vault.
Why not get rid of the client secret and use just the managed identity? The managed identity is for the application so it is not the same context as the user object, and while I do not know what the future brings MSAL and the MS Graph does not currently support this in our case. We could move to using client certificates which is a stronger authentication mechanism than a client secret, but this brings a complexity of its own that I don't want to bring in at this stage. So for now the answer is that we still need our secret, but we will keep it out of sight for all purposes to provide a "credentialless light" experience.
While we had a working deployment experience in part 2, with the caveats mentioned, we need to alter our deployment flow slightly to something like this:
- Create Key Vault
- Create App Registration and a secret.
- Store secret in Key Vault.
- Create Web App.
- Create Managed Identity for Web App.
- Assign permissions to Key Vault for the Managed Identity.
If you followed along in the previous post you might have noticed that the app registration parts were not in Bicep. They were provided in separate Azure CLI commands. So, how do we bring this all together if we want to stay with Bicep? There's two paths really:
- Do the Key Vault pieces in CLI commands as well.
- Create a layered Bicep deployment.
I prefer the second approach, but for completeness let's take a look at both options.
Creating Azure Key Vault in CLI
An Azure Key Vault can be created with Azure cli, and since the Azure AD App registration is also done this way we can combine the two:
- name: 'Create keyvault'
uses: azure/powershell@v1
with:
inlineScript: |
az group create --name "rg-${{ github.event.inputs.environment }}-kv-${{ github.event.inputs.appName }}" --location ${{ github.event.inputs.location }}
az keyvault create --name "kv-${{ github.event.inputs.appName }}-${{ github.run_number }}" --resource-group "rg-${{ github.event.inputs.environment }}-kv-${{ github.event.inputs.appName }}" --location ${{ github.event.inputs.location }} --enabled-for-template-deployment true
azPSVersion: "latest"
- name: Create Azure AD App Registration
id: appreg
run: |
export appName=${{ github.event.inputs.appName }}
appId=$(az ad app create --display-name $appName --query appId)
echo "::set-output name=appId::$appId"
- name: 'Generate ClientSecret & Store in Key Vault'
id: genClientSecret
uses: azure/powershell@v1
with:
inlineScript: |
($clientSecret = az ad app credential reset --credential-description appSecret --id ${{ steps.appreg.outputs.appId }} --query password) | az keyvault secret set --vault-name "kv-${{ github.event.inputs.appName }}-${{ github.run_number }}" --name clientSecret --value $clientSecret --output none
azPSVersion: "latest"
Link: https://github.com/ahelland/Bicep-Landing-Zones/blob/main/.github/workflows/DeployKeyVault.yml
This will indeed create a Key Vault and it will have a secret containing the app registration's clientSecret. So, that means we can just get the id of the key vault and feed into the Bicep for our Azure App Service? Not quite.
If we peek into the Azure Portal trying to verify the secret is there we notice something:
I am the owner of this Azure subscription, yet I cannot see the contents of the Key Vault. This is because Azure differentiates between control plane access and data plane access.
As an administrator I need to be able to manage the resource, so I can see the key vault is there. And if I for instance wanted to create a network level filter for who can reach the vault I am allowed to. These actions are about controlling the resource. However, the admin does not need to know the actual secrets a web app use - such actions are about the data of the resource.
With my permissions I can assign myself permissions to also have data plane access:
Going to the "Access policies" blade you can add yourself in the access list. You will notice that "GitHub Action" is there by default because the user account that creates the key vault is added automatically.
Ok, so let's just script in some more "stuff" and this will work? Well, yes, that is possible. However, "Access policies" is the legacy way for assigning permissions and you may have noted the screenshot shows an option for switching to "Azure Role-based Access Control". This is actually what we want to use.
This means that in addition to creating a key vault we need to create a "Role assignment" where the managed identity of the web app is granted a role that allows it to read secrets. Instead of adding on to our cli script let's just move to the second option for doing this 🙂
Create a layered Bicep deployment
Simple dependencies between resources is possible to handle in the Bicep code. For instance I need an App Service Plan before deploying the App Service on top of said plan. I've done this explicitly by using the dependsOn property, but there are also connections Bicep figures out entirely on its own.
This means that if I needed to have a key vault that the app used for storing values I could include it in the same Bicep and use an explicit depedency if the vault needed to be present before the app. In this case my dependency is a secret inside the vault so this doesn't work since the app registration is handled separately. I could generate the key vault and app service and the run the app registration, but then I would run into issues with values needed for configuring replyUrls in the app.
There isn't a magic button within Azure for handling these types of things, but Microsoft has recommendations regarding the architecture you use in Azure that will allow you to handle such scenarios. We are not able to dive into this here, but if you want more information take a look at the Cloud Adoption Framework (CAF) for more info:
https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/
Our takeaway from CAF here is that we configure a flow with multiple layers creating the key vault and doing the app registration in the first one, and creating the app service in the second using output from the first. (Note: CAF has recommendations for attaching specific resources to specific layers. We don't adhere to these in our sample so don't get confused when I use level 1 and 2 here.)
So, we create a key vault with RBAC enabled:
resource keyVault 'Microsoft.KeyVault/vaults@2021-11-01-preview' = {
name: name
location: location
tags: resourceTags
properties: {
sku: {
family: skuCode
name: sku
}
tenantId: tenantId
enableRbacAuthorization: true
}
}
Link: https://github.com/ahelland/Bicep-Landing-Zones/blob/main/modules/azure-key-vault/key-vault.bicep
Before generating a secret and attempting to store it you need to be aware of a minor snag - the user creating the vault does not get access to the data plane when using RBAC. The account used will not be able to assign roles to others either. To get around this we need to make sure the "GitHub Action" app registration is member of a custom role with these permissions. (The easy workaround is to make the app a member of the owners group of the subscription, but we want to reduce the scope.)
You will need a custom role definition like this:
{
"Name": "GitHub Action User",
"IsCustom": true,
"Description": "Grants permissions needed for CI/CD tasks.",
"Actions": [
"Microsoft.Authorization/roleAssignments/Delete",
"Microsoft.Authorization/roleAssignments/Write"
],
"NotActions": [],
"DataActions": [
"Microsoft.KeyVault/vaults/secrets/setSecret/action"
],
"NotDataActions": [],
"AssignableScopes": [
"/subscriptions/YOUR_SUBSCRIPTION_ID"
]
}
Apply it by running az role definition create --role-definition github_action_user_role.json.
Provided you followed my naming scheme (adapt if otherwise) assign like this:
$spObjectId=(az ad sp list --display-name "GitHub Action" --query [].objectId -o tsv)
az role assignment create --assignee $spObjectId --role "GitHub Action User"
To use key vault values in an app service you have to create key vault references instead of inserting the value itself. So, instead of clientSecret=foo we use the value '@Microsoft.KeyVault(VaultName=${keyVaultName};SecretName=clientSecret)'.
Enabling managed identity is just a manner of an extra line in the parameter list:
resource appservice 'Microsoft.Web/sites@2021-03-01' = {
name: name
location: location
tags: resourceTags
…
identity: {
type: 'SystemAssigned'
}
Built-in roles have predefined guids and the most appropriate role here would probably be "Key Vault Secrets User".
With this we can create a role assignment - this time in Bicep (where principalId is the id of the managed identity):
resource roleAssigment 'Microsoft.Authorization/roleAssignments@2020-10-01-preview' = {
name: name
scope: keyvault
properties: {
principalId: principalId
roleDefinitionId: '4633458b-17de-408a-b874-0445c86b69e6'
}
}
Note that the code above leaves out a few bits and pieces for readability - the actual code is spread across files and modularized and can be found here:
Stiching it all together we need to create a yaml file for GitHub Actions to run through. Just listing the individual steps here to illustrate the flow:
- name: Create Azure AD App Registration
id: appreg
run: |
…
- name: Deploy Azure Key Vault
id: deployKeyVault
uses: azure/cli@v1
with:
inlineScript: |
…
- name: 'Retrieve Key Vault suffix'
id: getKVOutputs
uses: azure/powershell@v1
with:
inlineScript: |
…
azPSVersion: "latest"
- name: 'Generate ClientSecret & Store in Key Vault'
id: genClientSecret
uses: azure/powershell@v1
with:
inlineScript: |
…
azPSVersion: "latest"
- name: 'Create Service Principal'
continue-on-error: true
run: |
…
- name: 'Add Permissions'
run: |
…
- name: 'Deploy Azure App Service'
id: deploy
uses: azure/cli@v1
with:
inlineScript: |
…
- name: 'Retrieve App Service Outputs'
id: getAppOutputs
uses: azure/powershell@v1
with:
inlineScript: |
…
azPSVersion: "latest"
- name: 'Update App Registration'
run: |
…
Look here for the complete Action including linting and what-if:
Yes, that took me longer than five minutes to assemble 🙂
If you head to the Azure Portal and click into your app service resource you should see something like this on the "Identity" blade:
And in the "Configuration" blade you will find something like this:
The end result is the same as we had in part 2, but you cannot read the secret by having access to the GitHub logs nor the Azure Portal. Of course as a developer you can certainly figure out ways to extract secrets, but that's another type of risk.
You can of course also assign managed identities to other resources, and a similar approach would work if you wanted separate components of an app to both be able to store files in a shared blob storage, and more. This post is more about following our web app sample to its logical conclusion.
But I don't want to use Azure App Services - is that where I'm stuck if I want to get rid of credentials? No, it's not - in the next part we will kick things up a notch and see what we can achieve when using Azure Kubernetes Service (AKS) as the runtime platform.
