Troubleshooting Guide: Resolving Azure App Services Image Pull Issues from Azure Container Registry

Welcome to our comprehensive guide dedicated to resolving challenges that arise when Azure App Services encounter difficulties pulling Docker images from the Azure Container Registry (ACR). Deploying applications seamlessly often hinges on a smooth image retrieval process, and encountering obstacles during this phase can disrupt the entire deployment workflow.

 

In this guide, we'll delve into a step-by-step checklist meticulously designed to diagnose and troubleshoot these issues effectively. From initial authentication, those include Admin Credentials, Managed Identity and access permissions to navigating complex network configurations for both Public and Private ingress on the ACR, this resource aims to be your go-to reference for untangling and resolving image pull issues within the Azure ecosystem.

 

Image and Tag:

 

Confirm that Image and Tag exist on the ACR and are spelled correctly:

 

Img. 1 - Image and Tag configuration on App Service, under Deployment Center

 

Img. 2 - Container Registry Repositories

 

 

Confirm the authentication type: Is the Web App trying to authenticate with Admin Credentials or Managed Identity (System Assigned/User Assigned)?

 

Admin Credentials:

• Ensure that Admin User is enabled on the Azure Container Registry (Registry authentication options - Azure Container Registry | Microsoft Learn (Opens in new window or tab))
• Ensure that the Web App has the DOCKER_REGISTRY_SERVER_USERNAME and DOCKER_REGISTRY_SERVER_PASSWORD App Settings with the correct credentials

Img. 3 - ACR Credentials and Admin User Setting

 

 

Img. 4 - App Service App Settings with ACR credentials

 

Managed Identity (System Assigned):

• Confirm if the Managed Identity is enabled and assigned to the Web App. (Managed identities - Azure App Service | Microsoft Learn (Opens in new window or tab))
• Confirm that the identity does have the AcrPull role assigned. (Authenticate with managed identity - Azure Container Registry | Microsoft Learn (Opens in new window or tab))

 

Managed Identity (User Assigned):

• Confirm if the Managed Identity is enabled and assigned to the Web App. (Managed identities - Azure App Service | Microsoft Learn (Opens in new window or tab))
• Confirm that the identity does have the AcrPull role assigned. (Authenticate with managed identity - Azure Container Registry | Microsoft Learn (Opens in new window or tab))

 

Img. 5 - AcrPull role assigned to Managed Identity (System assigned will have the same name of the App Service in question, User Assigned will have the name of the Managed Identity)

 

Check ACR network access: Is the Azure Container Registry using Public Access or Private Endpoints?

 

ACR Public Access:

• If Selected Networks option is enabled, confirm all the Web App Outbound IPs are added on the Firewall IPs (this can be further validated in the Web App Overview panel). This is not a proper solution, as the Web App outbound IPs may change eventually. Currently there is no other way to configure the Public Access using Selected Networks.
• Confirm if the Web App has the "Container image pull" property disabled and (if configured) WEBSITE_PULL_IMAGE_OVER_VNET App Setting disabled

 

Img. 6 - Vnet Configurations on App Service

 

 

Img. 7 - Container Image Pull property, should be disabled for public Access to the ACR

 

Img. 8 - App Setting as False

 

 

ACR Private Endpoint is enabled:

Note: This section involves several steps related to Networking configuration, for which is highly advised to proceed with the analysis with your Cloud Management/Networking team.

• Confirm if the Web App has the "Container image pull" property or WEBSITE_PULL_IMAGE_OVER_VNET App Setting enabled. (img 9 and 10)
• Confirm if the ACR Private Endpoint is in the same VNET as the Web App, or in a peered VNET.
• Check if the App Service VNET is using the default Azure DNS Server or a custom DNS Server: (Create, change, or delete an Azure virtual network | Microsoft Learn)
• If using the default Azure DNS Server: (Set up private endpoint with private link - Azure Container Registry | Microsoft Learn)
• Confirm if the VNET has Private DNS Zone enabled, with the name privatelink.azurecr.io
• Verify if the Private DNS Zone has been linked to the Web App VNET
• Confirm on Private DNS Zone if there are records pointing for the Private Endpoint IP
• If using a custom DNS Server:
• Confirm if the Web App has the WEBSITE_DNS_SERVER property set to the IP Address of the custom DNS Server. Otherwise, check if the VNET has the custom DNS Server configured as its DNS Server

 

 

Img. 9 - Container Image Pull property, should be enabled when using Private Endpoint

Img. 10 - App Service Setting configuring Pull image over Vnet

 

Test DNS resolution from Web app to ACR endpoint:

 

Linux Container:

• Open an SSH session for the Web App by browsing to the Kudu Console URL: https://<MY_WEB_APP>.scm.azurewebsites.net/newui/kududebug
• Once authenticated, you will see an in-browser shell, where you can run commands inside the Kudu sidecar container
• Run the following command to test the DNS resolution for the target ACR endpoint (<DNS_IP> if not using Custom DNS Server, please use: 168.63.129.16): nslookup <ACR_NAME>.azurecr.io <DNS_IP>
• If Container Registry does not have Private Endpoint, the IP resolved must be of the Public Range.
• If the Container Registry have a Private Endpoint configured (and only allows private traffic) the IP resolved must be a Private IP of the App Service Vnet Range.

Img. 11 - Example of nslookup working (the result does not need to be the same Address as on the image)

 

 

Windows Container:

For Windows Containers the SSH functionality will not be available if there is not a successfully running container. To do the above steps, please create a new Web App with the QuickStart image, under the same App Service Plan for the WebApp you are trying to configure (img 12).

 

Follow the same steps as above by browsing to the Windows Kudu Console URL: https://<MY_WEB_APP>.scm.azurewebsites.net/DebugConsole

Img. 12 - QuickStart image when creating new WebApp

 

Test TCP Ping connectivity to the ACR:

 

Linux Container:

Use tcpping on Kudu console to connect to the ACR endpoint on port 443: tcpping <ACR_NAME>.azurecr.io 443

If failing please check for NSG Rules that could be blocking this connection on both Subnets (App Service Subnet and ACR Private Endpoint Subnet)

Img. 13 - Tcpping working as expected

 

 

Windows Container:

For Windows Containers the SSH functionality will not be available if there is not a successfully running container. Similar to nslookup steps, please create a new Web App with the QuickStart image, under the same App Service Plan for the WebApp you are trying to configure (img 12).

 

Then you can use the command tcppingnative to connect to the ACR endpoint on port 443: tcppingnative <ACR_NAME>.azurecr.io 443

If failing please check for NSG Rules that could be blocking this connection on both Subnets (App Service Subnet and ACR Private Endpoint Subnet)

 

Img. 14 - tcppingnative working as expected

 

Side note: On Windows Containers depending if the base image is ServerCore or NanoServer the TCP connection troubleshooting command should be "tcpping" (for images that contain .NetFramework, like ServerCore) and "tcppingnative" (for images that does NOT contain .NetFramework, like NanoServer). Our default image from the QuickStart is using NanoServer so we need to use "tcppingnative".

 

 

 

Extra note:

In case you followed all of the checks mentioned on this guide and the issue still persists, please verify that your Image Manifest format is V2 schema 2 (Deprecated Engine Features | Docker Docs), since the V2 schema 1 is deprecated and when pulling image through Vnet it will fail. 

To confirm the Manifest version of your image, please review the steps at the following page: docker manifest | Docker Docs 

 

Img. 15: Example Manifest V2 schema 2

 

 

If you are still facing any unexpected constraints upon following the above configuration checks, please reach us out through a new Microsoft Support case, and we will gladly further assist you.