Public facing Azure Container Registry Reference Architecture

[Kumar-Ashwin-Hubert]

This reference architecture describes the deployment of secured Azure Container Registry for consuming docker images and artifacts by customer applications over external (public internet) network.

This architecture builds on Microsoft's recommended security best practices to expose private applications for external access. It utilizes the ACR's token and scope map feature to provide granular access control to ACR's repositories. Also, ACR internally uses the Docker APIs, and it is recommended to be familiar with these concepts before deploying this architecture. This reference architecture is developed to fulfill the following use cases:

1. Host container images for secured on-demand delivery of customer's application software and updates.
2. Restrict embargo countries/regions to download and deploy hosted container images.
3. Implement custom DNS solution for publicly accessible container registry.

Architecture

Components

This solution uses the following components:

• Azure Key Vault: Key Vault is provisioned for storing sensitive values and keys used specifically for on-premises systems like build pipeline and user registration application to access ACR API and function App to publish data to on-premises applications.
• Azure App Gateway: Azure App Gateway with WAF v2 is used to load balance and route traffic to Azure Container registry and provide Layer 7 connectivity to authenticate/authorize customer applications and download container images.
• Container registry: The container images for the workload are stored in a managed container registry. In this architecture, a single Azure Container Registry is used for all Kubernetes instances in the cluster. Geo-replication for Azure Container Registry enables replicating images to the selected Azure regions and providing continued access to images even if a region is experiencing an outage.
• Log Analytics: Regional Log Analytics instances are used for storing regional networking metrics and diagnostic logs. Additionally, a shared Log Analytics instance is used to store metrics and diagnostic logs for the ACR instance.
• Logging and threat detection: Microsoft Defender for Cloud built-in threat detection capability for Container Registry provides another layer of security intelligence. It detects unusual and potentially harmful attempts to access or exploit the Container Registry resources.
• Azure Functions: Azure Functions serves as the processing engine to manage the life cycle of container images in ACR. These functions also process specific logs and monitor security events and notify security and monitoring operations control via Azure Event Hubs.
• Kubernetes Cluster: Kubernetes Cluster is a consuming application cluster deployed by consumers of the hosted container images in Azure Container Registry.
• Hybrid Cloud Networking: Host the applications and tools (DevOps) used by developers to create container images and publish them to Azure Container Registry. Gateway Network is implemented to provide a secure communication layer between on-premises applications and applications hosted in Azure.

Azure Resources Configuration

When deploying this architecture, ensure that the following configurations are set properly.

Azure Container Registry

 

Overview

The Azure Container Registry (ACR) must be set to private access, so that all public requests flow through the Application Gateway.

Public access

The public network access should be disabled on the ACR to make the ACR secure and to allow public network access only via the App Gateway. In case there's a need to access the ACR directly via specific VNets, the public network access setting under Networking should be set to "Selected networks" or "Disabled" otherwise.

Private access

Make sure to create a private endpoint connection for the ACR and register it in a Private DNS zone. This configuration allows the app gateway to securely connect to the ACR.

Take note of the private endpoint FQDNs that are created, for instance contososecurepublicacr.privatelink.azurecr.io and contososecurepublicacr.eastus.data.privatelink.azurecr.io. These FQDNs are used while configuring the application gateway backend pool.

Azure Application Gateway

 

Overview

The end users/applications make requests to the app gateway, which internally routes them to Azure Container Registry.

To differentiate requests meant for the registry endpoint and the data endpoint, two custom domains (for example, acr-secure.contoso.com and acr-secure-data.contoso.com) must be set up to point to the same app gateway. The app gateway listeners are configured to listen to both endpoints. Based on the listener, the requests are privately routed over the virtual network to the corresponding backend pool of the app gateway that has the ACR's private endpoints configured.

ACR registry endpoint returns the authentication URL and the Data endpoint URL in the format *.azurecr.io as part of the REST endpoint response headers. However, to prevent end users from connecting directly to ACR, the application gateway's "Rewrite headers" capability is used to override the *.azurecr.io with *.contoso.com, ensuring that requests are routed through the custom domain and application gateway.

App Gateway Frontend IP Configuration

A public IP is needed for the app gateway to act as the front end for incoming requests. The custom domains, acr-secure.contoso.com and acr-secure-data.contoso.com in the example points to this public address of the app gateway. Create a new public IP address or use an existing public IP in the same location as the application gateway.

App Gateway Backend pools

The application gateway needs to be configured to have two backend pools – Registry and Data.

The backend targets for these pools point to the FQDN of the private links of ACR, such as contososecurepublicacr.privatelink.azurecr.io for the registry endpoint and contososecurepublicacr.eastus.data.privatelink.azurecr.io for the data endpoint.

Registry backend pool:

Data backend pool:

App Gateway Backend settings

Similarly, the backend settings need to be configured for both backend pools, ensuring that the backend protocol is set to HTTPS in order to achieve end-to-end TLS. Override the hostname with the ACR hostname so that the ACR certificate CN (Common Name) matches the request. Also, custom health probes are used which are covered in the next section.

Registry backend setting:

Data backend setting:

App Gateway Health probes

Custom health probes are used to determine the health of backend pools. Set the path to '/health' for both registry and data backends.

Registry backend health probe:

Data backend health probe:

App Gateway Listeners

In the app gateway, two HTTPS listeners need to be configured – one for Registry and another for Data. Custom hostnames, such as acr-secure.contoso.com for the registry and acr-secure-data.contoso.com for the data endpoint, are associated with these listeners. The listener type should be set to Multisite, and TLS/SSL certificates should be added during the configuration, as they handle HTTPS requests.

Registry listener:

Data listener:

App Gateway Rules

Routing rules in the app gateway are employed to route requests from the listeners to the corresponding backend targets. Two routing rules are required – one for the registry endpoint and one for the data endpoint, based on the listeners configured earlier.

Registry route rule:

Data route rule:

App Gateway Rewrites

Since ACR returns its own endpoints in various REST APIs, the application gateway's "rewrite headers" capability is used to replace the ACR endpoints with App gateway endpoints.

To achieve this, a rewrite set such as "acr-contoso-rewrite-set", must be created, and the following rewrite rules should be added specifically for the "registry routing rule"

1. Rewrite Location Header
2. Rewrite WWW-Authenticate Header
3. Rewrite Data Location Header

No rewrite rules should be added for the "data routing rule"

The rewrite rules to be configured for the "registry routing rule" are as follows:

1. Rewrite Location Header: 
   The 'Rewrite Location Header' rule rewrites the 'Location' header in the HTTP response that matches ACR endpoint to the registry/login endpoint of the app gateway when the status code is 302.
   • Add a condition to evaluate whether the location header in the response contains the Azure Container Registry endpoint,  contososecurepublicacr.azurecr.io in this example:
     1. Select Add condition and then select the box containing the If instructions to expand it.
     2. In the Type of variable to check list, select HTTP header.
     3. In the Header type list, choose Response Header.
     4. Select Common header under Header name.
     5. In the Common header list, select Location.
     6. Under Case-sensitive, select No.
     7. In the Operator list, select equal (=).
     8. Enter the regular expression pattern: https:\/\/contososecurepublicacr.azurecr.io(.*)$, and make sure to substitute 'contososecurepublicacr.azurecr.io' with your ACR instance endpoint.
     9. Select OK

        
   • Add an action to rewrite the location header:
     
     1. In the Rewrite type list, select Response Header.
     2. In the Action type list, select Set.
     3. Under Header name, select Common header.
     4. In the Common header list, choose Location.
     5. Use https://acr-secure.contoso.com{http_resp_Location_1} as the header value, and be sure to substitute 'acr-secure.contoso.com' with your custom domain endpoint. This action replaces contososecurepublicacr.azurecr.io with acr-secure.contoso.com in the location header.
     6. Select OK.

        
2. Rewrite WWW-Authenticate Header:
   The 'Rewrite WWW-Authenticate Header' rule modifies the 'WWW-Authenticate' header in the HTTP response that matches ACR endpoint to the registry/login endpoint of the app gateway when the status code is 401.

   • Add a condition to evaluate whether the WWW-Authenticate header in the response contains the Azure Container Registry endpoint, contososecurepublicacr.azurecr.io/oauth2/token in this case:
     1. Select Add condition and then chooses the box containing the If instructions to expand it.
     2. In the Type of variable to check list, select HTTP header.
     3. In the Header type list, choose Response.
     4. In the Header name, select Common header.
     5. In the Common header list, choose WWW-Authenticate.
     6. Under Case-sensitive, select No.
     7. In the Operator list, select equal (=).
     8. Enter the regular expression pattern: ^(.*)https:\/\/contososecurepublicacr.azurecr.io\/oauth2\/token(.*)$, and make sure to replace 'contososecurepublicacr.azurecr.io' in the pattern with your custom domain endpoint.
     9. Select OK

        
   • Add an action to rewrite the WWW-Authenticate header:
     1. In the Rewrite type list, select Response Header.
     2. In the Action type list, choose Set.
     3. Under Header name, select Common header.
     4. In the Common header list, choose WWW-Authenticate.
     5. Specify the header value: {http_resp_WWW-Authenticate_1}https://acr-secure.contoso.com/oauth2/token{http_resp_WWW-Authenticate_2}, and be sure to substitute 'acr-secure.contoso.com' with your custom domain endpoint. This action rule replaces contososecurepublicacr.azurecr.io with acr-secure.contoso.com in the WWW-Authenticate header.
     6. Select OK.

        
3. Rewrite Data Location Header:
   The 'Rewrite Data Location Header' rule rewrites the 'Location' header in the HTTP Response that matches the ACR data endpoint to the data endpoint of the app gateway when the status code is 302.
   • Add a condition to evaluate whether the location header in the response contains contososecurepublicacr.eastus.data.azurecr.io:
     1. Select Add condition and then choose the box containing the If instructions to expand it.
     2. In the Type of variable to check list, select HTTP header.
     3. In the Header type list, choose Response Header.
     4. Select Common header under Header name.
     5. In the Common header list, choose Location.
     6. Under Case-sensitive, select No.
     7. In the Operator list, select equal (=).
     8. Enter the regular expression pattern: https:\/\/contososecurepublicacr.eastus.data.azurecr.io(.*)$, and make sure to substitute 'contososecurepublicacr.eastus.data.azurecr.io' with your ACR instance data endpoint url.
     9. Select OK

        
   • Add an action to rewrite the location header:
     
     1. In the Rewrite type list, select Response Header.
     2. In the Action type list, choose Set.
     3. Under Header name, select Common header.
     4. In the Common header list, choose Location.
     5. Specify the header value: https://acr-secure-data.contoso.com{http_resp_Location_1}, and make sure to substitute 'acr-secure-data.contoso.com' with your custom domain data endpoint. This action effectively replaces contososecurepublicacr.eastus.data.azurecr.io with acr-secure-data.contoso.com in the location header.
     6. Select OK.

        

Azure Web Application Firewall (WAF)

Overview

Azure Web Application Firewall (WAF) on Azure Application Gateway provides the ability to Geo-filter traffic, permitting or blocking access from certain countries/regions to applications. This feature serves to restrict access to embargoed countries/regions or IPs identified by customer teams.

To configure WAF v2, either create or attach the WAF from the Web Application Firewall section of app gateway.

Within the Application Gateway WAF policy, custom rules can be implemented to block users based on their geo-location or IP address.

 

After testing, ensure that WAF is set to Prevention mode to block access that matches the rules.

Next Steps

This reference architecture depicts the integration of Azure Container Registry (ACR) and Azure Application Gateway with WAF policy to allow public access to the container images while keeping the ACR private. It also provides granular access control using the ACR's token and scope map feature. This architecture can be further extended to using geo-replication enabled ACR for multiple regions, by deploying an app gateway instance in each geo-replicated region and setting up a performance-based traffic manager with multiple app gateway endpoints.

Related resources

• Azure Container Registries Overview 
• ACR Authenticate with token 
• Azure Application Gateway Overview 
• Rewrite HTTP request and response headers with Azure Application Gateway 
• Azure Web Application Firewall (WAF) - Geomatch custom rules