Sometimes we get cases that are not usual, so it's always nice to document our findings, so that you may also investigate and solve your issues faster.
This article shows how Key Vault Policies may affect the functioning of a Logic App Standard, the troubleshooting steps and how to fix it.
Background
Logic App Standard works with many resources all together and as you know, it has a specific connection to the Storage Accounts and Fileshares, where it stores the main files and other supporting records (like in Tables and Queues).
If this connection is broken, the runtime will not be able to load as it cannot retrieve the needed files. So, when troubleshooting these types of errors, we usually check for the typical points of failure, like Private Endpoints and Vnet integrations.
Observed errors
While looking at a Logic App Standard (LA STD), we found a common error, that is the indicator of the connection severed to the Storage Account.
Our internal tools also showed that the Logic App was not able to connect to Storage Account (SA).
We knew that Vnet integration was enabled and that the Storage Account was behind Firewall (FW) and with Private Endpoints (PE).
As part of the troubleshooting, the first steps were to check the Storage Account Firewall and Private Endpoints:
- Storage Account Private Endpoint associated to Logic App Subnet
- Same region
- All 4 endpoints created (File, Blob, Table, Queue)
- All correctly configured
- Still the Logic App runtime was not loading
Network tests showed that Private Endpoint name resolution was occurring without issues, so DNS was not at fault.
TCPPing tests were also successful, from an external VM to the Storage Account.
So, when in doubt, attempt to revert to vanilla state, disable FW and PE from Storage Account and disable Vnet from Logic App. After this, the Logic App was still not loading.
Again, hostname resolution and TCPPing was OK, for the Storage endpoints.
Runtime internal Logs showed the following:
|
Details |
|
Microsoft.Azure.WebJobs.Script.WebHost.HttpException : Function host is not running. |
|
System.IO.IOException : The user name or password is incorrect. : 'C:\home\data\Functions\secrets\Sentinels' |
Log Stream showed the following:
This error is common and not very explicit. If we just cut off the Public Access, this message will also show. In fact, this image was obtained from one POC that I built, in which I had only cut off the Public Access.
As we wanted to recover the workflow files, to create a new Logic App, the user tried to browse the Logic App Fileshare. While doing this, our user was lacking the permissions to browse the Fileshare with the Microsoft Entra authentication type, so this was not valid.
Also, when trying to Browse using Access Key, this error was shown, blade failed to load with Key vault "unwrap key" error.
This was strange, as it's not usual to see this error.
Over at the backend logs side, Storage Operations Logs show similar issue.
Microsoft.WindowsAzure.Storage.StorageException: Unexpected HTTP status code 'Forbidden'.
---> System.Net.WebException: The remote server returned an error: (403) The key vault key is not found to unwrap the encryption key..
Findings
Deeper investigation showed that while the Encryption to the Storage Account was active and set to Customer Managed Keys (CMK), the Cryptographic operations were not allowed, thus not allowing Storage Account access using Access Key, as the keys were being stored in Key Vault for comparison.
From the documentation, we see that the Managed Identity is required to have minimum permissions, else it will fail to access the CMK.
Customer-managed keys for account encryption - Azure Storage | Microsoft Learn
"You can either create your own keys and store them in the key vault or managed HSM, or you can use the Azure Key Vault APIs to generate keys. The storage account and the key vault or managed HSM can be in different Microsoft Entra tenants, regions, and subscriptions.
…
The managed identity that is associated with the storage account must have these permissions at a minimum to access a customer-managed key in Azure Key Vault:
- wrapkey
- unwrapkey
- get
"
Conclusions and mitigation
Due to the Encryption of the Storage Keys and the restrictions on the Access Policies, Logic App Standard was not able to correctly connect to the Storage Account, resulting in a broken runtime.
As the Policies were enabled in the Key Vault, Storage Account access was reactivated from the Logic App side, and all succeeding tests were successful, including when enabling Vnet integration.