When working with Azure Function Apps on ILB ASE (Internal Load Balancing App Service Environment), one of the common errors we see is ”Azure Functions Runtime is unreachable”.
When this error occurs, the Azure portal points us to this doc, which lists the following potential causes for this error:
- Storage Account deleted
- Storage Account application settings deleted
- Storage Account credentials invalid
- Storage Account Inaccessible
Now, once you’ve isolated that the Storage account & application settings exist, and the credentials for it are in place – you’ll need to further investigate if the ILB ASE Function App can reach the Storage account.
First things first, ensure the firewall at the storage account level does not block access to the function App (having a firewall enabled at the Storage Account is not an issue – but you'll need to ensure it does not block the function app's accessing to it). So on the Storage Account's 'Firewalls and Virtual Networks' settings, you may either choose "Allow access from All networks" or you may choose "Allow access from Selected networks" and configure the ILB ASE's VNet & Subnet under the Virtual Networks section.
Once you ensure that the storage firewall isn't blocking the traffic, you can simply test connectivity from Function App using the Console tcpping to the storage account URL as follows.
>>tcpping <storageaccountname>.blob.core.windows.net
For an ILB ASE Function App, tcpping could sometimes lead to the following result:
The problem here is not quite intuitive. Capturing an F12 network trace while running a tcpping will give you a hint about the problem:
This simply means that when you use Console to tcpping, the console takes commands that you give it, calls Kudu’s REST API using the SCM endpoint, and then displays you the output in the console. Hence, access to the KUDU endpoint is essential.
- Although you can access the Azure portal from anywhere, to do a tcpping test for ILB ASE, you’ll need a machine that is either a VM in the same VNet as the ILB ASE, or from a machine has VPN’ed into the ILB ASE VNet, or from a machine that’s in a network connected to the ILB ASE VNet.
- Further, it is essential that your machine must be able to resolve to both the Function App hostname and Function App KUDU.
- To do this, you can either update the existing DNS Server accessible within that network, or simply add a hosts file entry at C:\Windows\System32\drivers\etc\hosts as follows:
Here, the IP 192.168.252.24 is the IP of my ILB ASE.
PS: So far, I haven’t found the need to reboot the machine after altering the hosts file.
I’ve seen customers promptly update the DNS to resolve to the Function App name but not to the KUDU, and this leaves them confused about the Console behavior during a tcpping.
Once the above settings are in place, the Console tcpping should be working fine:
If even after this point, the console tcpping to the storage endpoint fails, try accessing the KUDU site of the function App from the same machine. If you receive a certificate mismatch or Insecure message on the browser, it is likely to be caused due to the self-signed certificate that you may have uploaded on the ILB ASE. Self-signed certificate is accepted by an ILB ASE, but it is your responsibility to ensure that this certificate is installed on the Trusted Root CA store on your machine. If your machine does not trust this SSL certificate during an SSL Handshake, you may not be allowed to access KUDU site from that machine.
Here's a snippet of the Trusted Root CA certificates folder - this is where you'll need to install the ILB Certificate.
Once you've done this, the KUDU site access should work and tcpping to storage endpoint should be successful as well.
This also brings me touch upon a new change that has been introduced into ILB ASE's domain names. Newly created ILB ASEs will now have an extension of *.appserviceenvironment.net. This means that the certificate for this domain will be provided by Azure. This eliminates the need for a CA approved ILB Certificate or the need to install ILB's self-signed certificate in your machine's Trusted Root CA store.
For an ILB ASE with domain littleilb.appserviceenvironment.net and ILB IP 10.3.0.11 , the hosts file now looks like this:
Earlier, only the first 2 entries above were required. You need to add the last DNS entry (for littleilb.appserviceenvironment.net), otherwise the access to KUDU will fail with a DNS resolution error for the ILB Domain 😊