TheApplication Gateway for Containersis a new cutting-edge Azure service that offers load balancing and dynamic traffic management for applications running in a Kubernetes cluster. As part of Azure's Application Load Balancing portfolio, this innovative product provides an enhanced experience for developers and administrators. The Application Gateway for Containers represents the evolution of theApplication Gateway Ingress Controller (AGIC)and enables Azure Kubernetes Service (AKS) customers to leverage Azure's native Application Gateway load balancer. In this article, we will guide you through the process of deploying anAzure Kubernetes Service(AKS)cluster with anApplication Gateway for Containersin a fully automated fashion, using either a bring your own (BYO) or managed by ALB deployment.
API Server VNET Integrationallows you to enable network communication between the API server and the cluster nodes without requiring a private link or tunnel. AKS clusters with API Server VNET integration provide a series of advantages, for example, they can have public network access or private cluster mode enabled or disabled without redeploying the cluster. For more information, seeCreate an Azure Kubernetes Service cluster with API Server VNet Integration.
Azure NAT Gatewayto manage outbound connections initiated by AKS-hosted workloads.
Event-driven Autoscaling (KEDA) add-onis a single-purpose and lightweight component that strives to make application autoscaling simple and is a CNCF Incubation project.
Dapr extension for Azure Kubernetes Service (AKS)allows you to installDapr, a portable, event-driven runtime that simplifies building resilient, stateless, and stateful applications that run on the cloud and edge and embrace the diversity of languages and developer frameworks. With its sidecar architecture, Dapr helps you tackle the challenges that come with building microservices and keeps your code platform agnostic.
Vertical Pod Autoscalingallows you to automatically sets resource requests and limits on containers per workload based on past usage. VPA makes certain pods are scheduled onto nodes that have the required CPU and memory resources. For more information, seeKubernetes Vertical Pod Autoscaling.
Image Cleanerto clean up stale images on your Azure Kubernetes Service cluster.
Azure Kubernetes Service (AKS) Network Observabilityis an important part of maintaining a healthy and performant Kubernetes cluster. By collecting and analyzing data about network traffic, you can gain insights into how your cluster is operating and identify potential problems before they cause outages or performance degradation.
SystemSubnet: this subnet is used for the agent nodes of thesystemnode pool.
UserSubnet: this subnet is used for the agent nodes of theusernode pool.
PodSubnet: this subnet is used to allocate private IP addresses to pods dynamically.
ApiServerSubnet: API Server VNET Integration projects the API server endpoint directly into this delegated subnet in the virtual network where the AKS cluster is deployed.
AzureBastionSubnet: a subnet for the Azure Bastion Host.
VmSubnet: a subnet for a jump-box virtual machine used to connect to the (private) AKS cluster and for the private endpoints.
AppGwForConSubnet: this subnet contains the proxies created by the Application Load Balancer control plane to handle and distribute the ingress traffic to the AKS-hosted pods.
Microsoft.ServiceNetworking/trafficControllers: anApplication Gateway for Containersused as a service proxy to handle load balancing, routing, and TLS termination for AKS-hosted workloads. There are two deployment strategies for management of Application Gateway for Containers. You can decide specify the deployment strategy using theapplicationGatewayForContainersTypeparameter in themain.bicepmodule:
Bring your own (BYO) deployment: If you choose this strategy, the Bicep module creates the Application Gateway for Containers resource in the target deployment resource group. In this case, you are responsible to createAssociationandFrontendchild resources for the Application Gateway for Containers using the Azure Portal, Bicep, Azure CLI, Terraform, or Azure REST API. Every time you want to create a newGatewayor anIngressobject in your Azure Kubernetes Service (AKS) cluster, it's your responsibility to provision aFrontendchild resource for the Application Gateway for Containers upfront and reference it in the annotations in the Gateway or Ingress object. You are also responsible for deleting any Frontend child resource after deleting a Gateway or Ingress object in Kubernetes.
Managed by ALB Controller: In this deployment strategy Azure Load Balancer (ALB) Controller deployed in AKS using an Helm chart by the deployment script is responsible for the lifecycle of the Application Gateway for Containers resource and its sub resources. The ALB Controller creates Application Gateway for Containers resource in the AKS node resource group when anApplicationLoadBalancerKubernetes object is defined on the cluster. Every time you want to create a newGatewayor anIngressobject which references theApplicationLoadBalancerKubernetes object in the annotations, the ALB Controller provisions a new Frontend resource and manage its lifecycle based on the lifecycle of the Gateway or Ingress object.
Asystemnode pool in a dedicated subnet. The default node pool hosts only critical system pods and services. The worker nodes have node taint which prevents application pods from beings scheduled on this node pool.
Ausernode pool hosting user workloads and artifacts in a dedicated subnet.
Awindowsnode pool hosting Windows Server containers. This node pool is optionally created when the value of thewindowsAgentPoolEnabledequalstrue
Microsoft.Compute/virtualMachines: Bicep modules can optionally create a jump-box virtual machine to manage the private AKS cluster.
Microsoft.Network/bastionHosts: a separate Azure Bastion is deployed in the AKS cluster virtual network to provide SSH connectivity to both agent nodes and virtual machines.
Microsoft.Network/natGateways: a bring-your-own (BYO)Azure NAT Gatewayto manage outbound connections initiated by AKS-hosted workloads. The NAT Gateway is associated to theSystemSubnet,UserSubnet, andPodSubnetsubnets. TheoutboundTypeproperty of the cluster is set touserAssignedNatGatewayto specify that a BYO NAT Gateway is used for outbound connections. NOTE: you can update theoutboundTypeafter cluster creation and this will deploy or remove resources as required to put the cluster into the new egress configuration. For more information, seeUpdating outboundType after cluster creation.
Microsoft.Storage/storageAccounts: this storage account is used to store the boot diagnostics logs of both the service provider and service consumer virtual machines. Boot Diagnostics is a debugging feature that allows you to view console output and screenshots to diagnose virtual machine status.
Microsoft.ContainerRegistry/registries: an Azure Container Registry (ACR) to build, store, and manage container images and artifacts in a private registry for all container deployments.
Microsoft.Monitor/accounts: AnAzure Monitor workspaceis a unique environment for data collected byAzure Monitor. Each workspace has its own data repository, configuration, and permissions. Log Analytics workspaces contain logs and metrics data from multiple Azure resources, whereas Azure Monitor workspaces currently contain only metrics related toPrometheus. Azure Monitor managed service for Prometheus allows you to collect and analyze metrics at scale using a Prometheus-compatible monitoring solution, based on thePrometheus. This fully managed service allows you to use thePrometheus query language (PromQL)to analyze and alert on the performance of monitored infrastructure and workloads without having to operate the underlying infrastructure. The primary method for visualizing Prometheus metrics isAzure Managed Grafana. You can connect yourAzure Monitor workspaceto anAzure Managed Grafanato visualize Prometheus metrics using a set of built-in and custom Grafana dashboards.
The Bicep modules provide the flexibility to deploy the following Azure resources based on your requirements selectively:
Microsoft.CognitiveServices/accounts: anAzure OpenAI Servicewith aGPT-3.5model used by an AI application like a chatbot. Azure OpenAI Service gives customers advanced language AI with OpenAI GPT-4, GPT-3, Codex, and DALL-E models with Azure's security and enterprise promise. Azure OpenAI co-develops the APIs with OpenAI, ensuring compatibility and a smooth transition from one to the other.
NOTE You can find thearchitecture.vsdxfile used for the diagram under thevisiofolder.
What is Bicep?
Bicepis a domain-specific language (DSL) that uses a declarative syntax to deploy Azure resources. It provides concise syntax, reliable type safety, and support for code reuse. Bicep offers the best authoring experience for your infrastructure-as-code solutions in Azure.
What is Gateway API?
The Ingress resources Kubernetes objects have evolved into the more comprehensive and powerful Kubernetes Gateway API. Ingress Controller and Gateway API are both Kubernetes objects used for managing traffic routing and load balancing. While Ingress Controller served as entry points for external traffic, they had limitations in terms of flexibility and extensibility. The Kubernetes Gateway API emerged as a solution to address these limitations. Designed to be generic, expressive, extensible, and role-oriented, the Gateway API is a modern set of APIs for defining L4 and L7 routing rules in Kubernetes.
Gateway API offers superior functionality compared to Ingress Controllers as it separates listeners and routes into separate Kubernetes objects, Gateway and HTTPRoute. This separation allows different individuals with distinct roles and permissions to deploy them in separate namespaces. Additionally, Gateway API provides advanced traffic management capabilities including layer 7 HTTP/HTTPS request forwarding based on criteria such as hostname, path, headers, query string, methods, and ports. It also offers SSL termination and TLS policies for secure traffic management. These features grant better control and customization of traffic routing. The design of the Gateway API was driven by the following design goals to address and resolve issues and limitations in ingress controllers:
Role-oriented: The Gateway API comprises API resources that model organizational roles involved in using and configuring Kubernetes service networking.
Portable: Similar to Ingress, the Gateway API is designed to be a portable specification supported by multiple implementations.
Expressive: The Gateway API resources support core functionality such as header-based matching, traffic weighting, and other capabilities that were previously only possible through custom annotations in Ingress.
Extensible: The Gateway API allows for the linking of custom resources at different layers of the API, enabling granular customization within the API structure.
Additional notable capabilities of the Gateway API include:
GatewayClasses: Formalizes types of load-balancing implementations, making it easier for users to understand available capabilities through the Kubernetes resource model.
Shared Gateways and cross-Namespace support: Allows multiple Route resources to attach to the same Gateway, enabling load balancer and VIP sharing among teams and across Namespaces without direct coordination.
Typed Routes and typed backends: The Gateway API supports typed Route resources and different types of backends, providing flexibility in supporting various protocols (HTTP, gRPC) and backend targets (Kubernetes Services, storage buckets, functions).
ExperimentalService mesh supportwith the GAMMA initiative: The Gateway API enables the association of routing resources with Service resources, allowing the configuration of service meshes and ingress controllers.
Ingress Controllers are a straightforward option for setting up and are well-suited for smaller and less complex Kubernetes deployments that prioritize easy configuration.
If you currently have Ingress controllers configured in your Kubernetes cluster and they meet your requirements effectively, there may not be an immediate necessity to transition to the Kubernetes Gateway API.
Gateway APIis the recommended option in the following situations:
When dealing with complex routing configurations, traffic splitting, and advanced traffic management strategies, the flexibility provided by Kubernetes Gateway API's Route resources is essential.
In cases where networking requirements call for custom solutions or the integration of third-party plugins, the Kubernetes Gateway API's CRD-based approach offers enhanced extensibility.
What is Application Gateway for Containers?
TheApplication Gateway for Containersis a new cutting-edge Azure service that offers load balancing and dynamic traffic management for applications running in a Kubernetes cluster. As part of Azure's Application Load Balancing portfolio, this innovative product provides an enhanced experience for developers and administrators. The Application Gateway for Containers represents the evolution of theApplication Gateway Ingress Controller (AGIC)and enables Azure Kubernetes Service (AKS) customers to leverage Azure's native Application Gateway load balancer. Azure Application Gateway for Containers enables you to host multiple web applications on the same port, utilizing unique backend services. This allows for efficient multi-site hosting and simplifies the management of your containerized applications. The Application Gateway for Containers fully supports both theGateway APIandIngress APIKubernetes objects for traffic load balancing. For more information, see:
Azure Application Gateway for Containers supports two main deployment strategies:
Bring your own (BYO) deployment: If you choose this strategy, the Bicep module creates the Application Gateway for Containers resource in the target deployment resource group. In this case, you are responsible to createAssociationandFrontendchild resources for the Application Gateway for Containers using the Azure Portal, Bicep, Azure CLI, Terraform, or Azure REST API. Every time you want to create a newGatewayor anIngressobject in your Azure Kubernetes Service (AKS) cluster, it's your responsibility to provision aFrontendchild resource for the Application Gateway for Containers upfront and reference it in the annotations in the Gateway or Ingress object. After deleting a Gateway or Ingress object in Kubernetes, you are also responsible for deleting any Frontend child resource.
Managed by the Application Load Balancer (ALB) Controller: In this deployment strategy Azure Load Balancer (ALB) Controller deployed in AKS using an Helm chart by the deployment script is responsible for the lifecycle of the Application Gateway for Containers resource and its sub-resources. The ALB Controller creates an Application Gateway for Containers resource in the AKS node resource group when anApplicationLoadBalancerKubernetes object is defined on the cluster. Every time you want to create a newGatewayor anIngressobject which references theApplicationLoadBalancerKubernetes object in the annotations, the ALB Controller provisions a new Frontend resource and manage its lifecycle based on the lifecycle of the Gateway or Ingress object.
Application Gateway for Containers Components
The components of Azure Application Gateway for Containers include:
Core Components: Application Gateway for Containers is a parent Azure resource that manages the control plane, which handles the configuration and orchestration of proxies based on customer requirements. It serves as the parent resource for two important child resources: associations and frontends. These child resources are unique to each Application Gateway for Containers and cannot be shared with other instances of Application Gateway for Containers.
Frontend: An Application Gateway for Containers frontend is a sub-resource of the parent Application Gateway for Containers in Azure. It acts as the entry point for client traffic directed towards a specific Application Gateway for Containers. Each frontend is unique and cannot be associated with multiple Application Gateway for Containers. It provides a unique fully qualified domain name (FQDN) that can be assigned to a customer's CNAME record. Currently, private IP addresses are not supported for frontends. Additionally, a single Application Gateway for Containers has the ability to support multiple frontends.
Association: An Application Gateway for Containers association is a connection point into a virtual network and is a child resource of the Application Gateway for Containers. Application Gateway for Containers is designed to allow for multiple associations, but currently only one association is allowed. During the creation of an association, the necessary data plane is provisioned and connected to a subnet within the defined virtual network. Each association should have at least 256 available addresses in the subnet. If multiple Application Gateway for Containers are provisioned and each contains one association, the required number of available addresses should be n*256. It is important that all association resources match the same region as the parent Application Gateway for Containers resource. The subnet referenced by the association will contain the proxy components used to handle the ingress traffic to the Azure Kubernetes Service (AKS) cluster.
Managed identity: A user-defined managed identity with appropriate permissions must be provided for the ALB controller to update the control plane.
Application Load Balancer (ALB) Controller: The Application Gateway for Containers ALB Controller is a vital Kubernetes deployment that facilitates the seamless configuration and deployment of Application Gateway for Containers. By actively monitoring and responding to various Kubernetes Custom Resources and Resource configurations, such as Ingress, Gateway, and ApplicationLoadBalancer, the ALB Controller ensures efficient management of Application Gateway for Containers. Deployed using Helm, the ALB Controller comprises two essential pods. The first is the alb-controller pod, which takes charge of load balancing configuration for Application Gateway for Containers based on customer preferences and intent. The second is the alb-controller-bootstrap pod, responsible for effectively managing Custom Resource Definitions (CRDs) to further optimize the deployment process.
Managed proxies: These proxies route traffic directly to pods within your Azure Kubernetes Service (AKS) cluster. To ensure direct addressability, the cluster and the proxies must belong to the same virtual network and be configured with Azure CNI. The Application Load Balancer (ALB) control plane creates the proxies inside the subnet referenced by the association. For this reason, the user-defined managed identity used by the Application Load Balancer (ALB) Controller need to be assigned the Network Contributor role on this subnet. The latter needs to have at least with at least a /24 IP address space.
Features and Benefits
Azure Application Gateway for Containers offers a range of features and benefits, including:
Load Balancing: The service efficiently distributes incoming traffic across multiple containers, ensuring optimal performance and scalability. For more information, seeLoad balancing features.
Implementation of Gateway API: Application Gateway for Containers supports the Gateway API, which allows for the definition of routing rules and policies in a Kubernetes-native way. For more information, seeImplementation of Gateway API.
Custom Health Probe: You can define custom health probes to monitor the health of your containers and automatically route traffic away from unhealthy instances. For more information, seeCustom health probe for Application Gateway for Containers.
Session Affinity: The service provides session affinity, allowing you to maintain a consistent user experience by routing subsequent requests from the same client to the same container. For more information, seeApplication Gateway for Containers session affinity overview.
TLS Policy: Application Gateway for Containers supports TLS termination, allowing you to offload the SSL/TLS encryption and decryption process to the gateway. For more information, seeApplication Gateway for Containers TLS policy overview.
Header Rewrites: Application Gateway for Containers offers the capability to rewrite HTTP headers of client requests and responses from backend targets. Header Rewrites utilize theIngressExtensioncustom resource definition (CRD) of the Application Gateway for Containers. For more details, refer to the documentation on Header Rewrites forIngress APIandGateway API.
URL Rewrites: Application Gateway for Containers allows you to modify the URL of a client request, including the hostname and/or path. When Application Gateway for Containers initiates the request to the backend target, it includes the newly rewritten URL. Additional information on URL Rewrites can be found in the documentation forIngress APIandGateway API.
Advanced Load Balancing
The Application Gateway for Containers offers an impressive array of traffic management features to enhance your application deployment:
Layer 7 HTTP/HTTPS request forwarding capabilities based on prefix/exact match criteria, including hostname, path, headers, query strings, methods, and ports (80/443).
Robust support for HTTPS traffic management, including SSL termination and end-to-end SSL encryption.
Seamless integration with Ingress and Gateway API for streamlined configuration and management.
Flexible traffic splitting and weighted round-robin functionality to distribute traffic efficiently.
Mutual Authentication (mTLS) support for establishing secure connections to backend targets.
Robust health checks to ensure backends are healthy and capable of handling traffic before they are registered.
Automatic retries to optimize delivery of requests and handle potential failures gracefully.
TLS Policies that allow for granular control over the encryption protocols and ciphers used for secure communication.
Autoscaling capabilities to dynamically adjust resources based on workload demands.
Built-in resilience to handle availability zone failures and ensure continuous operation of your applications.
With these comprehensive features, the Application Gateway for Containers empowers you to efficiently manage and optimize your traffic flow. For more information, see For more information, seeLoad balancing features.
Tutorials and Samples
You can use the following tutorials to begin your journey with the Application Gateway for Containers:
You can find scripts and YAML manifests for the above tutorials under thetutorialsfolder.
Deploy the Bicep modules
You can deploy the Bicep modules in thebicepfolder using thedeploy.shBash script located in the same folder. Specify a value for the following parameters in thedeploy.shscript andmain.parameters.jsonparameters file before deploying the Bicep modules.
prefix: specifies a prefix for all the Azure resources.
authenticationType: specifies the type of authentication when accessing the Virtual Machine.sshPublicKeyis the recommended value. Allowed values:sshPublicKeyandpassword.
applicationGatewayForContainersType: this parameter specifies the deployment type for the Application Gateway for Containers:
managed: the Application Gateway for Containers resource and its child resources, association and frontends, are created and handled by the Azure Loab Balancer controller in the node resource group ofthe AKS cluster.
byo: the Application Gateway for Containers resource and its child resources are created in the targert resource group. You are responsible for the provisioning and deletion of the association and frontend child resources.
vmAdminUsername: specifies the name of the administrator account of the virtual machine.
vmAdminPasswordOrKey: specifies the SSH Key or password for the virtual machine.
aksClusterSshPublicKey: specifies the SSH Key or password for AKS cluster agent nodes.
aadProfileAdminGroupObjectIDs: when deploying an AKS cluster with Azure AD and Azure RBAC integration, this array parameter contains the list of Azure AD group object IDs that will have the admin role of the cluster.
keyVaultObjectIds: Specifies the object ID of the service principals to configure in Key Vault access policies.
windowsAgentPoolEnabled: Specifies whether to create a Windows Server agent pool.
The following table contains the code from theapplicationGatewayForContainers.bicepBicep module used to deploy aApplication Gateway for Containers.
// Parameters
@description('Specifies the name of the Application Gateway for Containers.')
param name string = 'dummy'
@description('Specifies whether the Application Gateway for Containers is managed or bring your own (BYO).')
@allowed([
'managed'
'byo'
])
param type string = 'managed'
@description('Specifies the workspace id of the Log Analytics used to monitor the Application Gateway for Containers.')
param workspaceId string
@description('Specifies the location of the Application Gateway for Containers.')
param location string
@description('Specifies the name of the existing AKS cluster.')
param aksClusterName string
@description('Specifies the name of the AKS cluster node resource group. This needs to be passed as a parameter and cannot be calculated inside this module.')
param nodeResourceGroupName string
@description('Specifies the name of the existing virtual network.')
param virtualNetworkName string
@description('Specifies the name of the subnet which contains the Application Gateway for Containers.')
param subnetName string
@description('Specifies the namespace for the Application Load Balancer Controller of the Application Gateway for Containers.')
param namespace string = 'azure-alb-system'
@description('Specifies the name of the service account for the Application Load Balancer Controller of the Application Gateway for Containers.')
param serviceAccountName string = 'alb-controller-sa'
@description('Specifies the resource tags for the Application Gateway for Containers.')
param tags object
// Variables
var diagnosticSettingsName = 'diagnosticSettings'
var logCategories = [
'TrafficControllerAccessLog'
]
var metricCategories = [
'AllMetrics'
]
var logs = [for category in logCategories: {
category: category
enabled: true
}]
var metrics = [for category in metricCategories: {
category: category
enabled: true
}]
// Resources
resource aksCluster 'Microsoft.ContainerService/managedClusters@2024-01-02-preview' existing = {
name: aksClusterName
}
resource virtualNetwork 'Microsoft.Network/virtualNetworks@2021-08-01' existing = {
name: virtualNetworkName
}
resource subnet 'Microsoft.Network/virtualNetworks/subnets@2021-08-01' existing = {
parent: virtualNetwork
name: subnetName
}
resource readerRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
name: 'acdd72a7-3385-48ef-bd42-f606fba81ae7'
scope: subscription()
}
resource networkContributorRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
name: '4d97b98b-1d4f-4787-a291-c67834d212e7'
scope: subscription()
}
resource appGwForContainersConfigurationManagerRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
name: 'fbc52c3f-28ad-4303-a892-8a056630b8f1'
scope: subscription()
}
resource applicationLoadBalancerManagedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2018-11-30' = {
name: '${name}ManagedIdentity'
location: location
tags: tags
}
// Assign the Network Contributor role to the Application Load Balancer user-assigned managed identity with the association subnet as as scope
resource subnetNetworkContributorRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
name: guid(name, applicationLoadBalancerManagedIdentity.name, networkContributorRole.id)
scope: subnet
properties: {
roleDefinitionId: networkContributorRole.id
principalId: applicationLoadBalancerManagedIdentity.properties.principalId
principalType: 'ServicePrincipal'
}
}
// Assign the AppGw for Containers Configuration Manager role to the Application Load Balancer user-assigned managed identity with the resource group as a scope
resource appGwForContainersConfigurationManagerRoleAssignmenOnResourceGroup 'Microsoft.Authorization/roleAssignments@2022-04-01' = if (type == 'byo') {
name: guid(name, applicationLoadBalancerManagedIdentity.name, appGwForContainersConfigurationManagerRole.id)
scope: resourceGroup()
properties: {
roleDefinitionId: appGwForContainersConfigurationManagerRole.id
principalId: applicationLoadBalancerManagedIdentity.properties.principalId
principalType: 'ServicePrincipal'
}
}
// Assign the AppGw for Containers Configuration Manager role to the Application Load Balancer user-assigned managed identity with the AKS cluster node resource group as a scope
module appGwForContainersConfigurationManagerRoleAssignmenOnnodeResourceGroupName 'resourceGroupRoleAssignment.bicep' = if (type == 'managed') {
name: guid(nodeResourceGroupName, applicationLoadBalancerManagedIdentity.name, appGwForContainersConfigurationManagerRole.id)
scope: resourceGroup(nodeResourceGroupName)
params: {
principalId: applicationLoadBalancerManagedIdentity.properties.principalId
roleName: appGwForContainersConfigurationManagerRole.name
}
}
// Assign the Reader role the Application Load Balancer user-assigned managed identity with the AKS cluster node resource group as a scope
module nodeResourceGroupReaderRoleAssignment 'resourceGroupRoleAssignment.bicep' = {
name: guid(nodeResourceGroupName, applicationLoadBalancerManagedIdentity.name, readerRole.id)
scope: resourceGroup(nodeResourceGroupName)
params: {
principalId: applicationLoadBalancerManagedIdentity.properties.principalId
roleName: readerRole.name
}
}
// Create federated identity for the Application Load Balancer user-assigned managed identity
resource federatedIdentityCredentials 'Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials@2023-01-31' = if (!empty(namespace) && !empty(serviceAccountName)) {
name: 'azure-alb-identity'
parent: applicationLoadBalancerManagedIdentity
properties: {
issuer: aksCluster.properties.oidcIssuerProfile.issuerURL
subject: 'system:serviceaccount:${namespace}:${serviceAccountName}'
audiences: [
'api://AzureADTokenExchange'
]
}
}
resource applicationGatewayForContainers 'Microsoft.ServiceNetworking/trafficControllers@2023-11-01' = if (type == 'byo') {
name: name
location: location
tags: tags
}
resource applicationGatewayDiagnosticSettings 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = if (type == 'byo') {
name: diagnosticSettingsName
scope: applicationGatewayForContainers
properties: {
workspaceId: workspaceId
logs: logs
metrics: metrics
}
}
// Outputs
output id string = applicationGatewayForContainers.id
output name string = applicationGatewayForContainers.name
output type string = applicationGatewayForContainers.type
output principalId string = applicationLoadBalancerManagedIdentity.properties.principalId
output clientId string = applicationLoadBalancerManagedIdentity.properties.clientId
The provided Bicep module performs the following steps:
Accepts several parameters, such as thename,type,location,tags, and more.
Defines variables for diagnostic settings, such asdiagnosticSettingsName,logCategories,metricCategories,logs, andmetrics.
Ceates a user-defined managed identity for the Application Load Balancer (ALB) Controller.
When thetypeparameter is set tobyo, creates an Application Gateway for Containers resource in the target resource group and sets up a diagnostics settings resource to collect logs and metrics from the Application Gateway for Containers in the specified Log Analytics workspace.
Assigns theNetwork Contributorrole to the Application Load Balancer user-assigned managed identity, scoped to the subnet.
When thetypeparameter is set tobyo, assigns theAppGw for Containers Configuration Managerrole to the Application Load Balancer user-assigned managed identity, scoped to the resource group. This role enables ALB Controller to access and configure the Application Gateway for Containers resource.
When thetypeparameter is set tomanaged, assigns theAppGw for Containers Configuration Managerrole to the Application Load Balancer user-assigned managed identity, scoped to the AKS cluster node resource group. In this case, the Application Gateway for Containers is created and managed by the ALB Controller in the AKS node resource group.
Assigns theReaderrole to the Application Load Balancer user-assigned managed identity, scoped to the AKS cluster node resource group.
Creates a federated identity credentials resource to establish a federated identity for the Application Load Balancer user-assigned managed identity. This is required by the ALB Controller and uses the nameazure-alb-identityfor the federated credential.
Creates anapplicationGatewayForContainersresource using the Microsoft.ServiceNetworking/trafficControllers resource type to create the Application Gateway for Containers based on the provided parameters.
Creates module outputs:
id,name, andtypeof the Application Gateway for Containers.
principalIdandclientIdof the ALB Controller user-defined managed identity.
When the value of thetypeparameter is set tobyo, the Bicep module creates an Application Gateway for Containers resource in the specified target resource group. Alternatively, when thetypeparameter is set tomanaged, the ALB Controller installed via Helm in the deployment script handles the creation and management of the Application Gateway for Containers in the AKS node resource group.
Deployment Script
The followingDeployment Scriptis used to run theinstall-alb-controller-sa.shBash script stofed in a public container of a storage container. This script installs necessary dependencies, retrieves cluster credentials, checks the cluster's type, installs Helm and Helm charts, creates namespaces and service accounts, and deploys the Application Load Balancer Controller.
# Install kubectl
az aks install-cli --only-show-errors
# Get AKS credentials
az aks get-credentials \
--admin \
--name $clusterName \
--resource-group $resourceGroupName \
--subscription $subscriptionId \
--only-show-errors
# Check if the cluster is private or not
private=$(az aks show --name $clusterName \
--resource-group $resourceGroupName \
--subscription $subscriptionId \
--query apiServerAccessProfile.enablePrivateCluster \
--output tsv)
# Install Helm
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 -o get_helm.sh -s
chmod 700 get_helm.sh
./get_helm.sh &>/dev/null
# Add Helm repos
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo add jetstack https://charts.jetstack.io
# Update Helm repos
helm repo update
# initialize variables
applicationGatewayForContainersName=''
diagnosticSettingName="DefaultDiagnosticSettings"
if [[ $private == 'true' ]]; then
# Log whether the cluster is public or private
echo "$clusterName AKS cluster is private"
# Install Prometheus
command="helm upgrade prometheus prometheus-community/kube-prometheus-stack \
--install \
--create-namespace \
--namespace prometheus \
--set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
--set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false"
az aks command invoke \
--name $clusterName \
--resource-group $resourceGroupName \
--subscription $subscriptionId \
--command "$command"
# Install NGINX ingress controller using the internal load balancer
command="helm upgrade nginx-ingress ingress-nginx/ingress-nginx \
--install \
--create-namespace \
--namespace ingress-basic \
--set controller.replicaCount=3 \
--set controller.nodeSelector.\"kubernetes\.io/os\"=linux \
--set defaultBackend.nodeSelector.\"kubernetes\.io/os\"=linux \
--set controller.metrics.enabled=true \
--set controller.metrics.serviceMonitor.enabled=true \
--set controller.metrics.serviceMonitor.additionalLabels.release=\"prometheus\" \
--set controller.service.annotations.\"service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path\"=/healthz"
az aks command invoke \
--name $clusterName \
--resource-group $resourceGroupName \
--subscription $subscriptionId \
--command "$command"
# Install certificate manager
command="helm upgrade cert-manager jetstack/cert-manager \
--install \
--create-namespace \
--namespace cert-manager \
--version v1.14.0 \
--set installCRDs=true \
--set nodeSelector.\"kubernetes\.io/os\"=linux \
--set \"extraArgs={--feature-gates=ExperimentalGatewayAPISupport=true}\""
az aks command invoke \
--name $clusterName \
--resource-group $resourceGroupName \
--subscription $subscriptionId \
--command "$command"
# Create cluster issuer
command="cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-nginx
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: $email
privateKeySecretRef:
name: letsencrypt
solvers:
- http01:
ingress:
class: nginx
podTemplate:
spec:
nodeSelector:
"kubernetes.io/os": linux
EOF"
az aks command invoke \
--name $clusterName \
--resource-group $resourceGroupName \
--subscription $subscriptionId \
--command "$command"
if [[ -n "$namespace" && \
-n "$serviceAccountName" ]]; then
# Create workload namespace
command="kubectl create namespace $namespace"
az aks command invoke \
--name $clusterName \
--resource-group $resourceGroupName \
--subscription $subscriptionId \
--command "$command"
# Create service account
command="cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
azure.workload.identity/client-id: $workloadManagedIdentityClientId
azure.workload.identity/tenant-id: $tenantId
labels:
azure.workload.identity/use: "true"
name: $serviceAccountName
namespace: $namespace
EOF"
az aks command invoke \
--name $clusterName \
--resource-group $resourceGroupName \
--subscription $subscriptionId \
--command "$command"
fi
if [[ "$applicationGatewayForContainersEnabled" == "true" \
&& -n "$applicationGatewayForContainersManagedIdentityClientId" \
&& -n "$applicationGatewayForContainersSubnetId" ]]; then
# Install the Application Load Balancer Controller
command="helm upgrade alb-controller oci://mcr.microsoft.com/application-lb/charts/alb-controller \
--install \
--create-namespace \
--namespace $applicationGatewayForContainersNamespace \
--version 1.0.0 \
--set albController.podIdentity.clientID=$applicationGatewayForContainersManagedIdentityClientId"
az aks command invoke \
--name $clusterName \
--resource-group $resourceGroupName \
--subscription $subscriptionId \
--command "$command"
# Create workload namespace
command="kubectl create namespace alb-infra"
az aks command invoke \
--name $clusterName \
--resource-group $resourceGroupName \
--subscription $subscriptionId \
--command "$command"
if [[ "$applicationGatewayForContainersType" == "managed" ]]; then
# Define the ApplicationLoadBalancer resource, specifying the subnet ID the Application Gateway for Containers association resource should deploy into.
# The association establishes connectivity from Application Gateway for Containers to the defined subnet (and connected networks where applicable) to
# be able to proxy traffic to a defined backend.
command="kubectl apply -f - <<EOF
apiVersion: alb.networking.azure.io/v1
kind: ApplicationLoadBalancer
metadata:
name: alb
namespace: alb-infra
spec:
associations:
- $applicationGatewayForContainersSubnetId
EOF"
az aks command invoke \
--name $clusterName \
--resource-group $resourceGroupName \
--subscription $subscriptionId \
--command "$command"
if [[ -n $nodeResourceGroupName ]]; then \
echo -n "Retrieving the resource id of the Application Gateway for Containers..."
counter=1
while [ $counter -le 600 ]
do
# Retrieve the resource id of the managed Application Gateway for Containers resource
applicationGatewayForContainersId=$(az resource list \
--resource-type "Microsoft.ServiceNetworking/TrafficControllers" \
--resource-group $nodeResourceGroupName \
--query [0].id \
--output tsv)
if [[ -n $applicationGatewayForContainersId ]]; then
echo
break
else
echo -n '.'
counter=$((counter + 1))
sleep 1
fi
done
if [[ -n $applicationGatewayForContainersId ]]; then
applicationGatewayForContainersName=$(basename $applicationGatewayForContainersId)
echo "[$applicationGatewayForContainersId] resource id of the [$applicationGatewayForContainersName] Application Gateway for Containers successfully retrieved"
else
echo "Failed to retrieve the resource id of the Application Gateway for Containers"
exit -1
fi
# Check if the diagnostic setting already exists for the Application Gateway for Containers
echo "Checking if the [$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers actually exists..."
result=$(az monitor diagnostic-settings show \
--name $diagnosticSettingName \
--resource $applicationGatewayForContainersId \
--query name \
--output tsv 2>/dev/null)
if [[ -z $result ]]; then
echo "[$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers does not exist"
echo "Creating [$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers..."
# Create the diagnostic setting for the Application Gateway for Containers
az monitor diagnostic-settings create \
--name $diagnosticSettingName \
--resource $applicationGatewayForContainersId \
--logs '[{"categoryGroup": "allLogs", "enabled": true}]' \
--metrics '[{"category": "AllMetrics", "enabled": true}]' \
--workspace $workspaceId \
--only-show-errors 1>/dev/null
if [[ $? == 0 ]]; then
echo "[$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers successfully created"
else
echo "Failed to create [$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers"
exit -1
fi
else
echo "[$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers already exists"
fi
fi
fi
fi
else
# Log whether the cluster is public or private
echo "$clusterName AKS cluster is public"
# Install Prometheus
echo "Installing Prometheus..."
helm upgrade prometheus prometheus-community/kube-prometheus-stack \
--install \
--create-namespace \
--namespace prometheus \
--set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
--set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false
if [[ $? == 0 ]]; then
echo "Prometheus successfully installed"
else
echo "Failed to install Prometheus"
exit -1
fi
# Install NGINX ingress controller using the internal load balancer
echo "Installing NGINX ingress controller..."
helm upgrade nginx-ingress ingress-nginx/ingress-nginx \
--install \
--create-namespace \
--namespace ingress-basic \
--set controller.replicaCount=3 \
--set controller.nodeSelector."kubernetes\.io/os"=linux \
--set defaultBackend.nodeSelector."kubernetes\.io/os"=linux \
--set controller.metrics.enabled=true \
--set controller.metrics.serviceMonitor.enabled=true \
--set controller.metrics.serviceMonitor.additionalLabels.release="prometheus" \
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz
if [[ $? == 0 ]]; then
echo "NGINX ingress controller successfully installed"
else
echo "Failed to install NGINX ingress controller"
exit -1
fi
# Install certificate manager
echo "Installing certificate manager..."
helm upgrade cert-manager jetstack/cert-manager \
--install \
--create-namespace \
--namespace cert-manager \
--version v1.14.0 \
--set installCRDs=true \
--set nodeSelector."kubernetes\.io/os"=linux \
--set "extraArgs={--feature-gates=ExperimentalGatewayAPISupport=true}"
if [[ $? == 0 ]]; then
echo "Certificate manager successfully installed"
else
echo "Failed to install certificate manager"
exit -1
fi
# Create cluster issuer
echo "Creating cluster issuer..."
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-nginx
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: $email
privateKeySecretRef:
name: letsencrypt
solvers:
- http01:
ingress:
class: nginx
podTemplate:
spec:
nodeSelector:
"kubernetes.io/os": linux
EOF
if [[ -n "$namespace" && \
-n "$serviceAccountName" ]]; then
# Create workload namespace
result=$(kubectl get namespace -o 'jsonpath={.items[?(@.metadata.name=="'$namespace'")].metadata.name'})
if [[ -n $result ]]; then
echo "$namespace namespace already exists in the cluster"
else
echo "$namespace namespace does not exist in the cluster"
echo "Creating $namespace namespace in the cluster..."
kubectl create namespace $namespace
fi
# Create service account
echo "Creating $serviceAccountName service account..."
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
azure.workload.identity/client-id: $workloadManagedIdentityClientId
azure.workload.identity/tenant-id: $tenantId
labels:
azure.workload.identity/use: "true"
name: $serviceAccountName
namespace: $namespace
EOF
fi
if [[ "$applicationGatewayForContainersEnabled" == "true" \
&& -n "$applicationGatewayForContainersManagedIdentityClientId" \
&& -n "$applicationGatewayForContainersSubnetId" ]]; then
# Install the Application Load Balancer
echo "Installing Application Load Balancer Controller in $applicationGatewayForContainersNamespace namespace using $applicationGatewayForContainersManagedIdentityClientId managed identity..."
helm upgrade alb-controller oci://mcr.microsoft.com/application-lb/charts/alb-controller \
--install \
--create-namespace \
--namespace $applicationGatewayForContainersNamespace \
--version 1.0.0 \
--set albController.namespace=$applicationGatewayForContainersNamespace \
--set albController.podIdentity.clientID=$applicationGatewayForContainersManagedIdentityClientId
if [[ $? == 0 ]]; then
echo "Application Load Balancer Controller successfully installed"
else
echo "Failed to install Application Load Balancer Controller"
exit -1
fi
if [[ "$applicationGatewayForContainersType" == "managed" ]]; then
# Create alb-infra namespace
albInfraNamespace='alb-infra'
result=$(kubectl get namespace -o 'jsonpath={.items[?(@.metadata.name=="'$albInfraNamespace'")].metadata.name'})
if [[ -n $result ]]; then
echo "$albInfraNamespace namespace already exists in the cluster"
else
echo "$albInfraNamespace namespace does not exist in the cluster"
echo "Creating $albInfraNamespace namespace in the cluster..."
kubectl create namespace $albInfraNamespace
fi
# Define the ApplicationLoadBalancer resource, specifying the subnet ID the Application Gateway for Containers association resource should deploy into.
# The association establishes connectivity from Application Gateway for Containers to the defined subnet (and connected networks where applicable) to
# be able to proxy traffic to a defined backend.
echo "Creating ApplicationLoadBalancer resource..."
kubectl apply -f - <<EOF
apiVersion: alb.networking.azure.io/v1
kind: ApplicationLoadBalancer
metadata:
name: alb
namespace: alb-infra
spec:
associations:
- $applicationGatewayForContainersSubnetId
EOF
if [[ -n $nodeResourceGroupName ]]; then \
echo -n "Retrieving the resource id of the Application Gateway for Containers..."
counter=1
while [ $counter -le 20 ]
do
# Retrieve the resource id of the managed Application Gateway for Containers resource
applicationGatewayForContainersId=$(az resource list \
--resource-type "Microsoft.ServiceNetworking/TrafficControllers" \
--resource-group $nodeResourceGroupName \
--query [0].id \
--output tsv)
if [[ -n $applicationGatewayForContainersId ]]; then
echo
break
else
echo -n '.'
counter=$((counter + 1))
sleep 1
fi
done
if [[ -n $applicationGatewayForContainersId ]]; then
applicationGatewayForContainersName=$(basename $applicationGatewayForContainersId)
echo "[$applicationGatewayForContainersId] resource id of the [$applicationGatewayForContainersName] Application Gateway for Containers successfully retrieved"
else
echo "Failed to retrieve the resource id of the Application Gateway for Containers"
exit -1
fi
# Check if the diagnostic setting already exists for the Application Gateway for Containers
echo "Checking if the [$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers actually exists..."
result=$(az monitor diagnostic-settings show \
--name $diagnosticSettingName \
--resource $applicationGatewayForContainersId \
--query name \
--output tsv 2>/dev/null)
if [[ -z $result ]]; then
echo "[$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers does not exist"
echo "Creating [$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers..."
# Create the diagnostic setting for the Application Gateway for Containers
az monitor diagnostic-settings create \
--name $diagnosticSettingName \
--resource $applicationGatewayForContainersId \
--logs '[{"categoryGroup": "allLogs", "enabled": true}]' \
--metrics '[{"category": "AllMetrics", "enabled": true}]' \
--workspace $workspaceId \
--only-show-errors 1>/dev/null
if [[ $? == 0 ]]; then
echo "[$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers successfully created"
else
echo "Failed to create [$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers"
exit -1
fi
else
echo "[$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers already exists"
fi
fi
fi
fi
fi
# Create output as JSON file
echo '{}' |
jq --arg x $applicationGatewayForContainersName '.applicationGatewayForContainersName=$x' |
jq --arg x $namespace '.namespace=$x' |
jq --arg x $serviceAccountName '.serviceAccountName=$x' |
jq --arg x 'prometheus' '.prometheus=$x' |
jq --arg x 'cert-manager' '.certManager=$x' |
jq --arg x 'ingress-basic' '.nginxIngressController=$x' >$AZ_SCRIPTS_OUTPUT_PATH
The script performs the following steps:
Installskubectlusing the Azure CLI commandaz aks install-cli.
Retrieves the AKS cluster credentials using the Azure CLI commandaz aks get-credentials.
Checks whether the AKS cluster is private or public by querying theenablePrivateClusterattribute of the cluster's API server access profile using the Azure CLI commandaz aks show.
InstallsHelmby downloading and executing theget_helm.shscript.
Updates the Helm repositories using thehelm repo updatecommand.
Initializes variables related to the Application Gateway for Containers.
The script performs the subsequent steps differently depending on whether the cluster is public or private. The script uses theaz aks command invoketo execute commands when the cluster is private.
Installs Prometheus using thehelm upgrade --installcommand.
Installs the certificate manager using thehelm upgrade --installcommand.
If anamespaceandserviceAccountNameare provided, it creates the namespace and service account usingkubectl. This information is optional, and it can be used to create the namespace and service account for a workload.
If the Application Gateway for Containers is enabled and necessary information is provided, it installs the Application Load Balancer Controller using thehelm upgrade --installcommand. The YAML manifest specifies the client id of the ALB Controller managed identity from theapplicationGatewayForContainersManagedIdentityClientIdenvironment variable and the target namespace from theapplicationGatewayForContainersNamespaceenvironment variable. For more information on the installation of the ALB Controller via Helm, seeQuickstart: Deploy Application Gateway for Containers ALB Controller.
When theapplicationGatewayForContainersTypeenvironment variable is set tomanaged, creates thealb-infranamespace usingkubectland deploys theApplicationLoadBalancerresource in the newly created namespace. The YAML manifest specifies the resource id of the subnet used by the association from theapplicationGatewayForContainersSubnetIdenvironment variable.
Retrieves the resource ID of the Application Gateway for Containers and checks if the diagnostic settings exist. If not, it creates the diagnostic settings usingaz monitor diagnostic-settings create.
Creates an output JSON file containing the Application Gateway for Containersname, worloadnamespaceandservice account name, if any, and the namespace for Prometheus and the Certificate Manager.
Review deployed resources
You can use the Azure portal to list the deployed resources in the resource group. If you chose to deploy an Application Gateway for Containers managed by the ALB Controller, you will find the resource under the node resource group of the AKS cluster.
You can also use Azure CLI to list the deployed resources in the resource group and AKS node resource group:
az resource list --resource-group <resource-group-name>
nodeResourceGroupName=$(az aks show --name <aks-name> --resource-group <resource-group-name> --query "nodeResourceGroup" -o tsv)
az resource list --resource-group nodeResourceGroupName
You can also use the following PowerShell cmdlet to list the deployed resources in the resource group and AKS node resource group:
After confirming the successful deployment, you can easily deploy your workloads and configure them to have a public endpoint through the newly created Application Gateway for Containers. To achieve this, you have two options: either aGatewayor anIngressin Kubernetes. These options allow you to expose your application to the public internet using the Application Gateway for Containers resource. The documentation provides various tutorials for both the Gateway API and the Ingress API, including:
You can find the scripts and YAML manifests for these tutorials in thetutorialsfolder. Additionally, theappfolder contains two samples:byofor a bring-your-own installation of the Application Gateway for Containers, andmanagedwhich works with Application Gateway for Containers managed by the ALB Controller.
For simplicity, let's focus on themanagedsample, while leaving thebyosample for the reader to review. Let's start by reviewing the YAML manifests.
Deployment
Thedeployment.yamlfile contains the YAML definition for the deployment, the service, and a secret that contains a temporary certificate for the Gatewy listener that will be replaced by the certificate issued byLet's Encryptvia the certificate manager. For more information on how to use the certificate manager to issue a new certificate to a Gateway using HTTP01 challenges, seeConfiguring the HTTP-01 Gateway API solver.
Thegateway.yamlcontains the definition of the Gateway used by the application. When using an Application Gateway for Containers managed by the ALB Controller, the frontend is auotmatically created for your by the ALB Controller.
The Gateway defines a certificate issuer in thecert-manager.io/issuerannotation, so we need to create an issuer. In the issuer, we define the CA root,Let's Encryptin this case, for the certificate chain to issue our certificate and the challenge type that our client would like to handle to prove our controll over the domain (in our case we will use HTTP01 challenge).
The certificate manager follows a series of steps to issue a certificate to the Gateway listener:
The gateway creates a Certificate object with a nil revision flag, initially pointing to a self-signed TLS secret. However, since the CA of the TLS secret is not valid, the Certificate realizes this and proceeds to the next step.
A CertificateRequest (CR) object is created with the revision flag set to 1, indicating the need to re-issue a valid certificate. The CR contains all the necessary information to send a CSR request in PKCS #10 format to the CA.
The CR object creates an Order object to monitor the request process.
The Cluster Issuer registers itself in the ACME server (CA) using our public key, which is included in the CR. The CA server generates a unique token/key for our request and associates them with our public key, enabling the verification of our signature for future requests from our client.
The CA server returns the unique token and key for each supported challenge to our client (Cluster Issuer), which were stored for our public key on the CA server.
The cluster issuer updates the Order object with the server-supported challenges, along with their unique token and key.
Based on the supported challenges and our Issuer configuration, the Order object determines to solve the HTTP01 Challenge, utilizing the parameters provided by the ACME server.
A new pod is created in the default namespace to run the Challenge. This pod contains an HTTP server that serves on a specific path and expects the origin to be our domain name. If the origin does not match, a 404 error is returned.
HTTPRoute
Thehttproute.yamlcontains the definition of theHTTPRouteobject used to route requests to the service:
Run the01-install-cert-manager.shscript if you need to install the certificate manager in your AKS cluster.
#/bin/bash
# Variables
source ./00-variables.sh
# Check if the cert-manager repository is not already added
result=$(helm repo list | grep $cmRepoName | awk '{print $1}')
if [[ -n $result ]]; then
echo "[$cmRepoName] Helm repo already exists"
else
# Add the Jetstack Helm repository
echo "Adding [$cmRepoName] Helm repo..."
helm repo add $cmRepoName $cmRepoUrl
fi
# Update your local Helm chart repository cache
echo 'Updating Helm repos...'
helm repo update
# Install cert-manager Helm chart
result=$(helm list -n $cmNamespace | grep $cmReleaseName | awk '{print $1}')
if [[ -n $result ]]; then
echo "[$cmReleaseName] cert-manager already exists in the $cmNamespace namespace"
echo "Upgrading [$cmReleaseName] cert-manager to the $cmNamespace namespace..."
else
# Install the cert-manager Helm chart
echo "Deploying [$cmReleaseName] cert-manager to the $cmNamespace namespace..."
fi
helm upgrade $cmReleaseName $cmRepoName/$cmChartName \
--install \
--create-namespace \
--namespace $cmNamespace \
--version $cmVersion \
--set installCRDs=true \
--set nodeSelector."kubernetes\.io/os"=linux \
--set "extraArgs={--feature-gates=ExperimentalGatewayAPISupport=true}"
Then run the02-create-sample.shscript to deploy the application to the specified namespace. The script makes use of theyqtool.
#!/bin/bash
# Variables
source ./00-variables.sh
# Check if namespace exists in the cluster
result=$(kubectl get namespace -o jsonpath="{.items[?(@.metadata.name=='$namespace')].metadata.name}")
if [[ -n $result ]]; then
echo "$namespace namespace already exists in the cluster"
else
echo "$namespace namespace does not exist in the cluster"
echo "creating $namespace namespace in the cluster..."
kubectl create namespace $namespace
fi
# Create a sample web application
kubectl apply -n $namespace -f ./deployment.yaml
# Create Gateway
cat gateway.yaml |
yq "(.metadata.name)|="\""$gatewayName"\" |
yq "(.metadata.namespace)|="\""$namespace"\" |
yq "(.metadata.annotations."\""cert-manager.io/issuer"\"")|="\""$issuerName"\" |
yq "(.metadata.annotations."\""alb.networking.azure.io/alb-name"\"")|="\""$applicationLoadBalancerName"\" |
yq "(.metadata.annotations."\""alb.networking.azure.io/alb-namespace"\"")|="\""$applicationLoadBalancerNamespace"\" |
yq "(.spec.listeners[1].hostname)|="\""$hostname"\" |
kubectl apply -f -
# Create Issuer
cat issuer.yaml |
yq "(.metadata.name)|="\""$issuerName"\" |
yq "(.metadata.namespace)|="\""$namespace"\" |
yq "(.spec.acme.solvers[0].http01.gatewayHTTPRoute.parentRefs[0].name)|="\""$gatewayName"\" |
yq "(.spec.acme.solvers[0].http01.gatewayHTTPRoute.parentRefs[0].namespace)|="\""$namespace"\" |
kubectl apply -f -
# Create HTTPRoute
cat httproute.yaml |
yq "(.metadata.name)|="\""$httpRouteName"\" |
yq "(.metadata.namespace)|="\""$namespace"\" |
yq "(.spec.parentRefs[0].name)|="\""$gatewayName"\" |
yq "(.spec.parentRefs[0].namespace)|="\""$namespace"\" |
kubectl apply -f -
If you delegated the management of your public DNS to Azure DNS, you can use the03-configure-dns.shscript to create a CNAME for the FQDN assigned to the frontend used by the Gateway.
# Variables
source ./00-variables.sh
# Get the FQDN of the gateway
echo -n "Retrieving the FQDN of the [$gatewayName] gateway..."
while true
do
fqdn=$(kubectl get gateway $gatewayName -n $namespace -o jsonpath='{.status.addresses[0].value}')
if [[ -n $fqdn ]]; then
echo
break
else
echo -n '.'
sleep 1
fi
done
if [ -n $fqdn ]; then
echo "[$fqdn] FQDN successfully retrieved from the [$gatewayName] gateway"
else
echo "Failed to retrieve the FQDN from the [$gatewayName] gateway"
exit
fi
# Check if an CNAME record for todolist subdomain exists in the DNS Zone
echo "Retrieving the CNAME for the [$subdomain] subdomain from the [$dnsZoneName] DNS zone..."
cname=$(az network dns record-set cname list \
--zone-name $dnsZoneName \
--resource-group $dnsZoneResourceGroupName \
--query "[?name=='$subdomain'].CNAMERecord.cname" \
--output tsv \
--only-show-errors)
if [[ -n $cname ]]; then
echo "A CNAME already exists in [$dnsZoneName] DNS zone for the [$subdomain]"
if [[ $cname == $fqdn ]]; then
echo "The [$cname] CNAME equals the FQDN of the [$gatewayName] gateway. No additional step is required."
exit
else
echo "The [$cname] CNAME is different than the [$fqdn] FQDN of the [$gatewayName] gateway"
fi
# Delete the CNAME record
echo "Deleting the [$subdomain] CNAME from the [$dnsZoneName] zone..."
az network dns record-set cname delete \
--name $subdomain \
--zone-name $dnsZoneName \
--resource-group $dnsZoneResourceGroupName \
--only-show-errors \
--yes
if [[ $? == 0 ]]; then
echo "[$subdomain] CNAME successfully deleted from the [$dnsZoneName] zone"
else
echo "Failed to delete the [$subdomain] CNAME from the [$dnsZoneName] zone"
exit
fi
else
echo "No CNAME exists in [$dnsZoneName] DNS zone for the [$subdomain] subdomain"
fi
# Create a CNAME record
echo "Creating a CNAME in the [$dnsZoneName] DNS zone for the [$fqdn] FQDN of the [$gatewayName] gateway..."
az network dns record-set cname set-record \
--cname $fqdn \
--zone-name $dnsZoneName \
--resource-group $dnsZoneResourceGroupName \
--record-set-name $subdomain \
--only-show-errors 1>/dev/null
if [[ $? == 0 ]]; then
echo "[$subdomain] CNAME successfully created in the [$dnsZoneName] DNS zone for the [$fqdn] FQDN of the [$gatewayName] gateway"
else
echo "Failed to create a CNAME in the [$dnsZoneName] DNS zone for the [$fqdn] FQDN of the [$gatewayName] gateway"
fi
Finally, you can test the sample by running the04-test-application.shscript.
#!/bin/bash
# Variables
source ./00-variables.sh
# Curling this FQDN should return responses from the backend as configured in the HTTPRoute
curl https://$hostname
You can also open the application using a web browser:
Clean up resources
You can delete the resource group using the following Azure CLI command when you no longer need the resources you created. This will remove all the Azure resources.
az group delete --name <resource-group-name>
Alternatively, you can use the following PowerShell cmdlet to delete the resource group and all the Azure resources.
Thank you paolosalvatori for Sharing this Awesome Blogpost with the Tech Community
Share
"}},"componentScriptGroups({\"componentId\":\"custom.widget.Social_Sharing\"})":{"__typename":"ComponentScriptGroups","scriptGroups":{"__typename":"ComponentScriptGroupsDefinition","afterInteractive":{"__typename":"PageScriptGroupDefinition","group":"AFTER_INTERACTIVE","scriptIds":[]},"lazyOnLoad":{"__typename":"PageScriptGroupDefinition","group":"LAZY_ON_LOAD","scriptIds":[]}},"componentScripts":[]},"component({\"componentId\":\"custom.widget.MicrosoftFooter\"})":{"__typename":"Component","render({\"context\":{\"component\":{\"entities\":[],\"props\":{}},\"page\":{\"entities\":[\"board:FastTrackforAzureBlog\",\"message:3967434\"],\"name\":\"BlogMessagePage\",\"props\":{},\"url\":\"https://techcommunity.microsoft.com/blog/fasttrackforazureblog/deploying-an-azure-kubernetes-service-aks-cluster-with-application-gateway-for-c/3967434\"}}})":{"__typename":"ComponentRenderResult","html":""}},"componentScriptGroups({\"componentId\":\"custom.widget.MicrosoftFooter\"})":{"__typename":"ComponentScriptGroups","scriptGroups":{"__typename":"ComponentScriptGroupsDefinition","afterInteractive":{"__typename":"PageScriptGroupDefinition","group":"AFTER_INTERACTIVE","scriptIds":[]},"lazyOnLoad":{"__typename":"PageScriptGroupDefinition","group":"LAZY_ON_LOAD","scriptIds":[]}},"componentScripts":[]},"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"components/community/NavbarDropdownToggle\"]})":[{"__ref":"CachedAsset:text:en_US-components/community/NavbarDropdownToggle-1737571274000"}],"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"shared/client/components/common/QueryHandler\"]})":[{"__ref":"CachedAsset:text:en_US-shared/client/components/common/QueryHandler-1737571274000"}],"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"components/messages/MessageCoverImage\"]})":[{"__ref":"CachedAsset:text:en_US-components/messages/MessageCoverImage-1737571274000"}],"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"shared/client/components/nodes/NodeTitle\"]})":[{"__ref":"CachedAsset:text:en_US-shared/client/components/nodes/NodeTitle-1737571274000"}],"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"components/messages/MessageTimeToRead\"]})":[{"__ref":"CachedAsset:text:en_US-components/messages/MessageTimeToRead-1737571274000"}],"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"components/messages/MessageSubject\"]})":[{"__ref":"CachedAsset:text:en_US-components/messages/MessageSubject-1737571274000"}],"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"components/users/UserLink\"]})":[{"__ref":"CachedAsset:text:en_US-components/users/UserLink-1737571274000"}],"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"shared/client/components/users/UserRank\"]})":[{"__ref":"CachedAsset:text:en_US-shared/client/components/users/UserRank-1737571274000"}],"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"components/messages/MessageTime\"]})":[{"__ref":"CachedAsset:text:en_US-components/messages/MessageTime-1737571274000"}],"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"components/messages/MessageBody\"]})":[{"__ref":"CachedAsset:text:en_US-components/messages/MessageBody-1737571274000"}],"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"components/messages/MessageCustomFields\"]})":[{"__ref":"CachedAsset:text:en_US-components/messages/MessageCustomFields-1737571274000"}],"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"components/messages/MessageRevision\"]})":[{"__ref":"CachedAsset:text:en_US-components/messages/MessageRevision-1737571274000"}],"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"components/messages/MessageReplyButton\"]})":[{"__ref":"CachedAsset:text:en_US-components/messages/MessageReplyButton-1737571274000"}],"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"components/messages/MessageAuthorBio\"]})":[{"__ref":"CachedAsset:text:en_US-components/messages/MessageAuthorBio-1737571274000"}],"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"shared/client/components/users/UserAvatar\"]})":[{"__ref":"CachedAsset:text:en_US-shared/client/components/users/UserAvatar-1737571274000"}],"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"shared/client/components/ranks/UserRankLabel\"]})":[{"__ref":"CachedAsset:text:en_US-shared/client/components/ranks/UserRankLabel-1737571274000"}],"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"components/users/UserRegistrationDate\"]})":[{"__ref":"CachedAsset:text:en_US-components/users/UserRegistrationDate-1737571274000"}],"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"shared/client/components/nodes/NodeAvatar\"]})":[{"__ref":"CachedAsset:text:en_US-shared/client/components/nodes/NodeAvatar-1737571274000"}],"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"shared/client/components/nodes/NodeDescription\"]})":[{"__ref":"CachedAsset:text:en_US-shared/client/components/nodes/NodeDescription-1737571274000"}],"message({\"id\":\"message:3984515\"})":{"__ref":"BlogReplyMessage:message:3984515"},"message({\"id\":\"message:3968544\"})":{"__ref":"BlogReplyMessage:message:3968544"},"message({\"id\":\"message:3967479\"})":{"__ref":"BlogReplyMessage:message:3967479"},"message({\"id\":\"message:3967457\"})":{"__ref":"BlogReplyMessage:message:3967457"},"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"components/tags/TagView/TagViewChip\"]})":[{"__ref":"CachedAsset:text:en_US-components/tags/TagView/TagViewChip-1737571274000"}],"cachedText({\"lastModified\":\"1737571274000\",\"locale\":\"en-US\",\"namespaces\":[\"shared/client/components/nodes/NodeIcon\"]})":[{"__ref":"CachedAsset:text:en_US-shared/client/components/nodes/NodeIcon-1737571274000"}]},"CachedAsset:pages-1742487242464":{"__typename":"CachedAsset","id":"pages-1742487242464","value":[{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"BlogViewAllPostsPage","type":"BLOG","urlPath":"/category/:categoryId/blog/:boardId/all-posts/(/:after|/:before)?","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"CasePortalPage","type":"CASE_PORTAL","urlPath":"/caseportal","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"CreateGroupHubPage","type":"GROUP_HUB","urlPath":"/groups/create","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"CaseViewPage","type":"CASE_DETAILS","urlPath":"/case/:caseId/:caseNumber","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"InboxPage","type":"COMMUNITY","urlPath":"/inbox","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"HelpFAQPage","type":"COMMUNITY","urlPath":"/help","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"IdeaMessagePage","type":"IDEA_POST","urlPath":"/idea/:boardId/:messageSubject/:messageId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"IdeaViewAllIdeasPage","type":"IDEA","urlPath":"/category/:categoryId/ideas/:boardId/all-ideas/(/:after|/:before)?","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"LoginPage","type":"USER","urlPath":"/signin","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"BlogPostPage","type":"BLOG","urlPath":"/category/:categoryId/blogs/:boardId/create","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"UserBlogPermissions.Page","type":"COMMUNITY","urlPath":"/c/user-blog-permissions/page","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"ThemeEditorPage","type":"COMMUNITY","urlPath":"/designer/themes","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"TkbViewAllArticlesPage","type":"TKB","urlPath":"/category/:categoryId/kb/:boardId/all-articles/(/:after|/:before)?","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1730819800000,"localOverride":null,"page":{"id":"AllEvents","type":"CUSTOM","urlPath":"/Events","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"OccasionEditPage","type":"EVENT","urlPath":"/event/:boardId/:messageSubject/:messageId/edit","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"OAuthAuthorizationAllowPage","type":"USER","urlPath":"/auth/authorize/allow","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"PageEditorPage","type":"COMMUNITY","urlPath":"/designer/pages","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"PostPage","type":"COMMUNITY","urlPath":"/category/:categoryId/:boardId/create","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"ForumBoardPage","type":"FORUM","urlPath":"/category/:categoryId/discussions/:boardId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"TkbBoardPage","type":"TKB","urlPath":"/category/:categoryId/kb/:boardId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"EventPostPage","type":"EVENT","urlPath":"/category/:categoryId/events/:boardId/create","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"UserBadgesPage","type":"COMMUNITY","urlPath":"/users/:login/:userId/badges","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"GroupHubMembershipAction","type":"GROUP_HUB","urlPath":"/membership/join/:nodeId/:membershipType","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"MaintenancePage","type":"COMMUNITY","urlPath":"/maintenance","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"IdeaReplyPage","type":"IDEA_REPLY","urlPath":"/idea/:boardId/:messageSubject/:messageId/comments/:replyId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"UserSettingsPage","type":"USER","urlPath":"/mysettings/:userSettingsTab","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"GroupHubsPage","type":"GROUP_HUB","urlPath":"/groups","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"ForumPostPage","type":"FORUM","urlPath":"/category/:categoryId/discussions/:boardId/create","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"OccasionRsvpActionPage","type":"OCCASION","urlPath":"/event/:boardId/:messageSubject/:messageId/rsvp/:responseType","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"VerifyUserEmailPage","type":"USER","urlPath":"/verifyemail/:userId/:verifyEmailToken","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"AllOccasionsPage","type":"OCCASION","urlPath":"/category/:categoryId/events/:boardId/all-events/(/:after|/:before)?","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"EventBoardPage","type":"EVENT","urlPath":"/category/:categoryId/events/:boardId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"TkbReplyPage","type":"TKB_REPLY","urlPath":"/kb/:boardId/:messageSubject/:messageId/comments/:replyId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"IdeaBoardPage","type":"IDEA","urlPath":"/category/:categoryId/ideas/:boardId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"CommunityGuideLinesPage","type":"COMMUNITY","urlPath":"/communityguidelines","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"CaseCreatePage","type":"SALESFORCE_CASE_CREATION","urlPath":"/caseportal/create","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"TkbEditPage","type":"TKB","urlPath":"/kb/:boardId/:messageSubject/:messageId/edit","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"ForgotPasswordPage","type":"USER","urlPath":"/forgotpassword","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"IdeaEditPage","type":"IDEA","urlPath":"/idea/:boardId/:messageSubject/:messageId/edit","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"TagPage","type":"COMMUNITY","urlPath":"/tag/:tagName","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"BlogBoardPage","type":"BLOG","urlPath":"/category/:categoryId/blog/:boardId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"OccasionMessagePage","type":"OCCASION_TOPIC","urlPath":"/event/:boardId/:messageSubject/:messageId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"ManageContentPage","type":"COMMUNITY","urlPath":"/managecontent","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"ClosedMembershipNodeNonMembersPage","type":"GROUP_HUB","urlPath":"/closedgroup/:groupHubId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"CommunityPage","type":"COMMUNITY","urlPath":"/","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"ForumMessagePage","type":"FORUM_TOPIC","urlPath":"/discussions/:boardId/:messageSubject/:messageId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"IdeaPostPage","type":"IDEA","urlPath":"/category/:categoryId/ideas/:boardId/create","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1730819800000,"localOverride":null,"page":{"id":"CommunityHub.Page","type":"CUSTOM","urlPath":"/Directory","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"BlogMessagePage","type":"BLOG_ARTICLE","urlPath":"/blog/:boardId/:messageSubject/:messageId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"RegistrationPage","type":"USER","urlPath":"/register","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"EditGroupHubPage","type":"GROUP_HUB","urlPath":"/group/:groupHubId/edit","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"ForumEditPage","type":"FORUM","urlPath":"/discussions/:boardId/:messageSubject/:messageId/edit","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"ResetPasswordPage","type":"USER","urlPath":"/resetpassword/:userId/:resetPasswordToken","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1730819800000,"localOverride":null,"page":{"id":"AllBlogs.Page","type":"CUSTOM","urlPath":"/blogs","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"TkbMessagePage","type":"TKB_ARTICLE","urlPath":"/kb/:boardId/:messageSubject/:messageId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"BlogEditPage","type":"BLOG","urlPath":"/blog/:boardId/:messageSubject/:messageId/edit","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"ManageUsersPage","type":"USER","urlPath":"/users/manage/:tab?/:manageUsersTab?","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"ForumReplyPage","type":"FORUM_REPLY","urlPath":"/discussions/:boardId/:messageSubject/:messageId/replies/:replyId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"PrivacyPolicyPage","type":"COMMUNITY","urlPath":"/privacypolicy","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"NotificationPage","type":"COMMUNITY","urlPath":"/notifications","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"UserPage","type":"USER","urlPath":"/users/:login/:userId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"OccasionReplyPage","type":"OCCASION_REPLY","urlPath":"/event/:boardId/:messageSubject/:messageId/comments/:replyId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"ManageMembersPage","type":"GROUP_HUB","urlPath":"/group/:groupHubId/manage/:tab?","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"SearchResultsPage","type":"COMMUNITY","urlPath":"/search","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"BlogReplyPage","type":"BLOG_REPLY","urlPath":"/blog/:boardId/:messageSubject/:messageId/replies/:replyId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"GroupHubPage","type":"GROUP_HUB","urlPath":"/group/:groupHubId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"TermsOfServicePage","type":"COMMUNITY","urlPath":"/termsofservice","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"CategoryPage","type":"CATEGORY","urlPath":"/category/:categoryId","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"ForumViewAllTopicsPage","type":"FORUM","urlPath":"/category/:categoryId/discussions/:boardId/all-topics/(/:after|/:before)?","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"TkbPostPage","type":"TKB","urlPath":"/category/:categoryId/kbs/:boardId/create","__typename":"PageDescriptor"},"__typename":"PageResource"},{"lastUpdatedTime":1742487242464,"localOverride":null,"page":{"id":"GroupHubPostPage","type":"GROUP_HUB","urlPath":"/group/:groupHubId/:boardId/create","__typename":"PageDescriptor"},"__typename":"PageResource"}],"localOverride":false},"CachedAsset:text:en_US-components/context/AppContext/AppContextProvider-0":{"__typename":"CachedAsset","id":"text:en_US-components/context/AppContext/AppContextProvider-0","value":{"noCommunity":"Cannot find community","noUser":"Cannot find current user","noNode":"Cannot find node with id {nodeId}","noMessage":"Cannot find message with id {messageId}"},"localOverride":false},"CachedAsset:text:en_US-shared/client/components/common/Loading/LoadingDot-0":{"__typename":"CachedAsset","id":"text:en_US-shared/client/components/common/Loading/LoadingDot-0","value":{"title":"Loading..."},"localOverride":false},"User:user:-1":{"__typename":"User","id":"user:-1","uid":-1,"login":"Deleted","email":"","avatar":null,"rank":null,"kudosWeight":1,"registrationData":{"__typename":"RegistrationData","status":"ANONYMOUS","registrationTime":null,"confirmEmailStatus":false,"registrationAccessLevel":"VIEW","ssoRegistrationFields":[]},"ssoId":null,"profileSettings":{"__typename":"ProfileSettings","dateDisplayStyle":{"__typename":"InheritableStringSettingWithPossibleValues","key":"layout.friendly_dates_enabled","value":"false","localValue":"true","possibleValues":["true","false"]},"dateDisplayFormat":{"__typename":"InheritableStringSetting","key":"layout.format_pattern_date","value":"MMM dd yyyy","localValue":"MM-dd-yyyy"},"language":{"__typename":"InheritableStringSettingWithPossibleValues","key":"profile.language","value":"en-US","localValue":"en","possibleValues":["en-US"]}},"deleted":false},"Theme:customTheme1":{"__typename":"Theme","id":"customTheme1"},"Category:category:FastTrack":{"__typename":"Category","id":"category:FastTrack","entityType":"CATEGORY","displayId":"FastTrack","nodeType":"category","depth":3,"title":"Microsoft FastTrack","shortTitle":"Microsoft FastTrack","parent":{"__ref":"Category:category:products-services"}},"Category:category:top":{"__typename":"Category","id":"category:top","displayId":"top","nodeType":"category","depth":0,"title":"Top","entityType":"CATEGORY","shortTitle":"Top"},"Category:category:communities":{"__typename":"Category","id":"category:communities","displayId":"communities","nodeType":"category","depth":1,"parent":{"__ref":"Category:category:top"},"title":"Communities","entityType":"CATEGORY","shortTitle":"Communities"},"Category:category:products-services":{"__typename":"Category","id":"category:products-services","displayId":"products-services","nodeType":"category","depth":2,"parent":{"__ref":"Category:category:communities"},"title":"Products","entityType":"CATEGORY","shortTitle":"Products"},"Blog:board:FastTrackforAzureBlog":{"__typename":"Blog","id":"board:FastTrackforAzureBlog","entityType":"BLOG","displayId":"FastTrackforAzureBlog","nodeType":"board","depth":4,"conversationStyle":"BLOG","title":"FastTrack for Azure","description":"","avatar":null,"profileSettings":{"__typename":"ProfileSettings","language":null},"parent":{"__ref":"Category:category:FastTrack"},"ancestors":{"__typename":"CoreNodeConnection","edges":[{"__typename":"CoreNodeEdge","node":{"__ref":"Community:community:gxcuf89792"}},{"__typename":"CoreNodeEdge","node":{"__ref":"Category:category:communities"}},{"__typename":"CoreNodeEdge","node":{"__ref":"Category:category:products-services"}},{"__typename":"CoreNodeEdge","node":{"__ref":"Category:category:FastTrack"}}]},"userContext":{"__typename":"NodeUserContext","canAddAttachments":false,"canUpdateNode":false,"canPostMessages":false,"isSubscribed":false},"boardPolicies":{"__typename":"BoardPolicies","canPublishArticleOnCreate":{"__typename":"PolicyResult","failureReason":{"__typename":"FailureReason","message":"error.lithium.policies.forums.policy_can_publish_on_create_workflow_action.accessDenied","key":"error.lithium.policies.forums.policy_can_publish_on_create_workflow_action.accessDenied","args":[]}}},"shortTitle":"FastTrack for Azure","repliesProperties":{"__typename":"RepliesProperties","sortOrder":"REVERSE_PUBLISH_TIME","repliesFormat":"threaded"},"eventPath":"category:FastTrack/category:products-services/category:communities/community:gxcuf89792board:FastTrackforAzureBlog/","tagProperties":{"__typename":"TagNodeProperties","tagsEnabled":{"__typename":"PolicyResult","failureReason":null}},"requireTags":true,"tagType":"PRESET_ONLY"},"AssociatedImage:{\"url\":\"https://techcommunity.microsoft.com/t5/s/gxcuf89792/images/cmstNC05WEo0blc\"}":{"__typename":"AssociatedImage","url":"https://techcommunity.microsoft.com/t5/s/gxcuf89792/images/cmstNC05WEo0blc","height":512,"width":512,"mimeType":"image/png"},"Rank:rank:4":{"__typename":"Rank","id":"rank:4","position":6,"name":"Microsoft","color":"333333","icon":{"__ref":"AssociatedImage:{\"url\":\"https://techcommunity.microsoft.com/t5/s/gxcuf89792/images/cmstNC05WEo0blc\"}"},"rankStyle":"OUTLINE"},"User:user:988334":{"__typename":"User","id":"user:988334","uid":988334,"login":"paolosalvatori","deleted":false,"avatar":{"__typename":"UserAvatar","url":"https://techcommunity.microsoft.com/t5/s/gxcuf89792/images/dS05ODgzMzQtMzg1MjYyaTE4QTU5MkIyQUVCMkM0MDE"},"rank":{"__ref":"Rank:rank:4"},"email":"","messagesCount":67,"biography":null,"topicsCount":30,"kudosReceivedCount":159,"kudosGivenCount":29,"kudosWeight":1,"registrationData":{"__typename":"RegistrationData","status":null,"registrationTime":"2021-03-05T07:56:49.951-08:00","confirmEmailStatus":null},"followersCount":null,"solutionsCount":0,"entityType":"USER","eventPath":"community:gxcuf89792/user:988334"},"BlogTopicMessage:message:3967434":{"__typename":"BlogTopicMessage","uid":3967434,"subject":"Deploying an Azure Kubernetes Service (AKS) Cluster with Application Gateway for Containers","id":"message:3967434","revisionNum":12,"repliesCount":4,"author":{"__ref":"User:user:988334"},"depth":0,"hasGivenKudo":false,"board":{"__ref":"Blog:board:FastTrackforAzureBlog"},"conversation":{"__ref":"Conversation:conversation:3967434"},"messagePolicies":{"__typename":"MessagePolicies","canPublishArticleOnEdit":{"__typename":"PolicyResult","failureReason":{"__typename":"FailureReason","message":"error.lithium.policies.forums.policy_can_publish_on_edit_workflow_action.accessDenied","key":"error.lithium.policies.forums.policy_can_publish_on_edit_workflow_action.accessDenied","args":[]}},"canModerateSpamMessage":{"__typename":"PolicyResult","failureReason":{"__typename":"FailureReason","message":"error.lithium.policies.feature.moderation_spam.action.moderate_entity.allowed.accessDenied","key":"error.lithium.policies.feature.moderation_spam.action.moderate_entity.allowed.accessDenied","args":[]}}},"contentWorkflow":{"__typename":"ContentWorkflow","state":"PUBLISH","scheduledPublishTime":null,"scheduledTimezone":null,"userContext":{"__typename":"MessageWorkflowContext","canSubmitForReview":null,"canEdit":false,"canRecall":null,"canSubmitForPublication":null,"canReturnToAuthor":null,"canPublish":null,"canReturnToReview":null,"canSchedule":false},"shortScheduledTimezone":null},"readOnly":false,"editFrozen":false,"moderationData":{"__ref":"ModerationData:moderation_data:3967434"},"teaser":"
This article provides a detailed guide on installing an AKS (Azure Kubernetes Service) cluster and Azure Application Gateway for Containers, all using Bicep.
\n
\n
\n
","body":"
TheApplication Gateway for Containersis a new cutting-edge Azure service that offers load balancing and dynamic traffic management for applications running in a Kubernetes cluster. As part of Azure's Application Load Balancing portfolio, this innovative product provides an enhanced experience for developers and administrators. The Application Gateway for Containers represents the evolution of theApplication Gateway Ingress Controller (AGIC)and enables Azure Kubernetes Service (AKS) customers to leverage Azure's native Application Gateway load balancer. In this article, we will guide you through the process of deploying anAzure Kubernetes Service(AKS)cluster with anApplication Gateway for Containersin a fully automated fashion, using either a bring your own (BYO) or managed by ALB deployment.
API Server VNET Integrationallows you to enable network communication between the API server and the cluster nodes without requiring a private link or tunnel. AKS clusters with API Server VNET integration provide a series of advantages, for example, they can have public network access or private cluster mode enabled or disabled without redeploying the cluster. For more information, seeCreate an Azure Kubernetes Service cluster with API Server VNet Integration.
\n
Azure NAT Gatewayto manage outbound connections initiated by AKS-hosted workloads.
\n
Event-driven Autoscaling (KEDA) add-onis a single-purpose and lightweight component that strives to make application autoscaling simple and is a CNCF Incubation project.
\n
Dapr extension for Azure Kubernetes Service (AKS)allows you to installDapr, a portable, event-driven runtime that simplifies building resilient, stateless, and stateful applications that run on the cloud and edge and embrace the diversity of languages and developer frameworks. With its sidecar architecture, Dapr helps you tackle the challenges that come with building microservices and keeps your code platform agnostic.
Vertical Pod Autoscalingallows you to automatically sets resource requests and limits on containers per workload based on past usage. VPA makes certain pods are scheduled onto nodes that have the required CPU and memory resources. For more information, seeKubernetes Vertical Pod Autoscaling.
Image Cleanerto clean up stale images on your Azure Kubernetes Service cluster.
\n
Azure Kubernetes Service (AKS) Network Observabilityis an important part of maintaining a healthy and performant Kubernetes cluster. By collecting and analyzing data about network traffic, you can gain insights into how your cluster is operating and identify potential problems before they cause outages or performance degradation.
SystemSubnet: this subnet is used for the agent nodes of thesystemnode pool.
\n
UserSubnet: this subnet is used for the agent nodes of theusernode pool.
\n
PodSubnet: this subnet is used to allocate private IP addresses to pods dynamically.
\n
ApiServerSubnet: API Server VNET Integration projects the API server endpoint directly into this delegated subnet in the virtual network where the AKS cluster is deployed.
\n
AzureBastionSubnet: a subnet for the Azure Bastion Host.
\n
VmSubnet: a subnet for a jump-box virtual machine used to connect to the (private) AKS cluster and for the private endpoints.
\n
AppGwForConSubnet: this subnet contains the proxies created by the Application Load Balancer control plane to handle and distribute the ingress traffic to the AKS-hosted pods.
\n
\n
\n
Microsoft.ServiceNetworking/trafficControllers: anApplication Gateway for Containersused as a service proxy to handle load balancing, routing, and TLS termination for AKS-hosted workloads. There are two deployment strategies for management of Application Gateway for Containers. You can decide specify the deployment strategy using theapplicationGatewayForContainersTypeparameter in themain.bicepmodule:\n
\n
Bring your own (BYO) deployment: If you choose this strategy, the Bicep module creates the Application Gateway for Containers resource in the target deployment resource group. In this case, you are responsible to createAssociationandFrontendchild resources for the Application Gateway for Containers using the Azure Portal, Bicep, Azure CLI, Terraform, or Azure REST API. Every time you want to create a newGatewayor anIngressobject in your Azure Kubernetes Service (AKS) cluster, it's your responsibility to provision aFrontendchild resource for the Application Gateway for Containers upfront and reference it in the annotations in the Gateway or Ingress object. You are also responsible for deleting any Frontend child resource after deleting a Gateway or Ingress object in Kubernetes.
\n
Managed by ALB Controller: In this deployment strategy Azure Load Balancer (ALB) Controller deployed in AKS using an Helm chart by the deployment script is responsible for the lifecycle of the Application Gateway for Containers resource and its sub resources. The ALB Controller creates Application Gateway for Containers resource in the AKS node resource group when anApplicationLoadBalancerKubernetes object is defined on the cluster. Every time you want to create a newGatewayor anIngressobject which references theApplicationLoadBalancerKubernetes object in the annotations, the ALB Controller provisions a new Frontend resource and manage its lifecycle based on the lifecycle of the Gateway or Ingress object.
Asystemnode pool in a dedicated subnet. The default node pool hosts only critical system pods and services. The worker nodes have node taint which prevents application pods from beings scheduled on this node pool.
\n
Ausernode pool hosting user workloads and artifacts in a dedicated subnet.
\n
Awindowsnode pool hosting Windows Server containers. This node pool is optionally created when the value of thewindowsAgentPoolEnabledequalstrue
Microsoft.Compute/virtualMachines: Bicep modules can optionally create a jump-box virtual machine to manage the private AKS cluster.
\n
Microsoft.Network/bastionHosts: a separate Azure Bastion is deployed in the AKS cluster virtual network to provide SSH connectivity to both agent nodes and virtual machines.
\n
Microsoft.Network/natGateways: a bring-your-own (BYO)Azure NAT Gatewayto manage outbound connections initiated by AKS-hosted workloads. The NAT Gateway is associated to theSystemSubnet,UserSubnet, andPodSubnetsubnets. TheoutboundTypeproperty of the cluster is set touserAssignedNatGatewayto specify that a BYO NAT Gateway is used for outbound connections. NOTE: you can update theoutboundTypeafter cluster creation and this will deploy or remove resources as required to put the cluster into the new egress configuration. For more information, seeUpdating outboundType after cluster creation.
\n
Microsoft.Storage/storageAccounts: this storage account is used to store the boot diagnostics logs of both the service provider and service consumer virtual machines. Boot Diagnostics is a debugging feature that allows you to view console output and screenshots to diagnose virtual machine status.
\n
Microsoft.ContainerRegistry/registries: an Azure Container Registry (ACR) to build, store, and manage container images and artifacts in a private registry for all container deployments.
Microsoft.Monitor/accounts: AnAzure Monitor workspaceis a unique environment for data collected byAzure Monitor. Each workspace has its own data repository, configuration, and permissions. Log Analytics workspaces contain logs and metrics data from multiple Azure resources, whereas Azure Monitor workspaces currently contain only metrics related toPrometheus. Azure Monitor managed service for Prometheus allows you to collect and analyze metrics at scale using a Prometheus-compatible monitoring solution, based on thePrometheus. This fully managed service allows you to use thePrometheus query language (PromQL)to analyze and alert on the performance of monitored infrastructure and workloads without having to operate the underlying infrastructure. The primary method for visualizing Prometheus metrics isAzure Managed Grafana. You can connect yourAzure Monitor workspaceto anAzure Managed Grafanato visualize Prometheus metrics using a set of built-in and custom Grafana dashboards.
The Bicep modules provide the flexibility to deploy the following Azure resources based on your requirements selectively:
\n
\n
Microsoft.CognitiveServices/accounts: anAzure OpenAI Servicewith aGPT-3.5model used by an AI application like a chatbot. Azure OpenAI Service gives customers advanced language AI with OpenAI GPT-4, GPT-3, Codex, and DALL-E models with Azure's security and enterprise promise. Azure OpenAI co-develops the APIs with OpenAI, ensuring compatibility and a smooth transition from one to the other.
NOTE You can find thearchitecture.vsdxfile used for the diagram under thevisiofolder.
\n
\n
\n
What is Bicep?
\n
Bicepis a domain-specific language (DSL) that uses a declarative syntax to deploy Azure resources. It provides concise syntax, reliable type safety, and support for code reuse. Bicep offers the best authoring experience for your infrastructure-as-code solutions in Azure.
\n
\n
What is Gateway API?
\n
The Ingress resources Kubernetes objects have evolved into the more comprehensive and powerful Kubernetes Gateway API. Ingress Controller and Gateway API are both Kubernetes objects used for managing traffic routing and load balancing. While Ingress Controller served as entry points for external traffic, they had limitations in terms of flexibility and extensibility. The Kubernetes Gateway API emerged as a solution to address these limitations. Designed to be generic, expressive, extensible, and role-oriented, the Gateway API is a modern set of APIs for defining L4 and L7 routing rules in Kubernetes.
\n
\n
\n
\n
Gateway API offers superior functionality compared to Ingress Controllers as it separates listeners and routes into separate Kubernetes objects, Gateway and HTTPRoute. This separation allows different individuals with distinct roles and permissions to deploy them in separate namespaces. Additionally, Gateway API provides advanced traffic management capabilities including layer 7 HTTP/HTTPS request forwarding based on criteria such as hostname, path, headers, query string, methods, and ports. It also offers SSL termination and TLS policies for secure traffic management. These features grant better control and customization of traffic routing. The design of the Gateway API was driven by the following design goals to address and resolve issues and limitations in ingress controllers:
\n
\n
\n
Role-oriented: The Gateway API comprises API resources that model organizational roles involved in using and configuring Kubernetes service networking.
\n
Portable: Similar to Ingress, the Gateway API is designed to be a portable specification supported by multiple implementations.
\n
Expressive: The Gateway API resources support core functionality such as header-based matching, traffic weighting, and other capabilities that were previously only possible through custom annotations in Ingress.
\n
Extensible: The Gateway API allows for the linking of custom resources at different layers of the API, enabling granular customization within the API structure.
\n
\n
Additional notable capabilities of the Gateway API include:
\n
\n
GatewayClasses: Formalizes types of load-balancing implementations, making it easier for users to understand available capabilities through the Kubernetes resource model.
\n
Shared Gateways and cross-Namespace support: Allows multiple Route resources to attach to the same Gateway, enabling load balancer and VIP sharing among teams and across Namespaces without direct coordination.
\n
Typed Routes and typed backends: The Gateway API supports typed Route resources and different types of backends, providing flexibility in supporting various protocols (HTTP, gRPC) and backend targets (Kubernetes Services, storage buckets, functions).
\n
ExperimentalService mesh supportwith the GAMMA initiative: The Gateway API enables the association of routing resources with Service resources, allowing the configuration of service meshes and ingress controllers.
Ingress Controllers are a straightforward option for setting up and are well-suited for smaller and less complex Kubernetes deployments that prioritize easy configuration.
\n
If you currently have Ingress controllers configured in your Kubernetes cluster and they meet your requirements effectively, there may not be an immediate necessity to transition to the Kubernetes Gateway API.
\n
\n
Gateway APIis the recommended option in the following situations:
\n
\n
When dealing with complex routing configurations, traffic splitting, and advanced traffic management strategies, the flexibility provided by Kubernetes Gateway API's Route resources is essential.
\n
In cases where networking requirements call for custom solutions or the integration of third-party plugins, the Kubernetes Gateway API's CRD-based approach offers enhanced extensibility.
\n
\n
What is Application Gateway for Containers?
\n
TheApplication Gateway for Containersis a new cutting-edge Azure service that offers load balancing and dynamic traffic management for applications running in a Kubernetes cluster. As part of Azure's Application Load Balancing portfolio, this innovative product provides an enhanced experience for developers and administrators. The Application Gateway for Containers represents the evolution of theApplication Gateway Ingress Controller (AGIC)and enables Azure Kubernetes Service (AKS) customers to leverage Azure's native Application Gateway load balancer. Azure Application Gateway for Containers enables you to host multiple web applications on the same port, utilizing unique backend services. This allows for efficient multi-site hosting and simplifies the management of your containerized applications. The Application Gateway for Containers fully supports both theGateway APIandIngress APIKubernetes objects for traffic load balancing. For more information, see:
Azure Application Gateway for Containers supports two main deployment strategies:
\n
\n
\n
Bring your own (BYO) deployment: If you choose this strategy, the Bicep module creates the Application Gateway for Containers resource in the target deployment resource group. In this case, you are responsible to createAssociationandFrontendchild resources for the Application Gateway for Containers using the Azure Portal, Bicep, Azure CLI, Terraform, or Azure REST API. Every time you want to create a newGatewayor anIngressobject in your Azure Kubernetes Service (AKS) cluster, it's your responsibility to provision aFrontendchild resource for the Application Gateway for Containers upfront and reference it in the annotations in the Gateway or Ingress object. After deleting a Gateway or Ingress object in Kubernetes, you are also responsible for deleting any Frontend child resource.
\n
Managed by the Application Load Balancer (ALB) Controller: In this deployment strategy Azure Load Balancer (ALB) Controller deployed in AKS using an Helm chart by the deployment script is responsible for the lifecycle of the Application Gateway for Containers resource and its sub-resources. The ALB Controller creates an Application Gateway for Containers resource in the AKS node resource group when anApplicationLoadBalancerKubernetes object is defined on the cluster. Every time you want to create a newGatewayor anIngressobject which references theApplicationLoadBalancerKubernetes object in the annotations, the ALB Controller provisions a new Frontend resource and manage its lifecycle based on the lifecycle of the Gateway or Ingress object.
\n
\n
\n
Application Gateway for Containers Components
\n
The components of Azure Application Gateway for Containers include:
\n
\n
\n
Core Components: Application Gateway for Containers is a parent Azure resource that manages the control plane, which handles the configuration and orchestration of proxies based on customer requirements. It serves as the parent resource for two important child resources: associations and frontends. These child resources are unique to each Application Gateway for Containers and cannot be shared with other instances of Application Gateway for Containers.
\n
Frontend: An Application Gateway for Containers frontend is a sub-resource of the parent Application Gateway for Containers in Azure. It acts as the entry point for client traffic directed towards a specific Application Gateway for Containers. Each frontend is unique and cannot be associated with multiple Application Gateway for Containers. It provides a unique fully qualified domain name (FQDN) that can be assigned to a customer's CNAME record. Currently, private IP addresses are not supported for frontends. Additionally, a single Application Gateway for Containers has the ability to support multiple frontends.
\n
Association: An Application Gateway for Containers association is a connection point into a virtual network and is a child resource of the Application Gateway for Containers. Application Gateway for Containers is designed to allow for multiple associations, but currently only one association is allowed. During the creation of an association, the necessary data plane is provisioned and connected to a subnet within the defined virtual network. Each association should have at least 256 available addresses in the subnet. If multiple Application Gateway for Containers are provisioned and each contains one association, the required number of available addresses should be n*256. It is important that all association resources match the same region as the parent Application Gateway for Containers resource. The subnet referenced by the association will contain the proxy components used to handle the ingress traffic to the Azure Kubernetes Service (AKS) cluster.
\n
Managed identity: A user-defined managed identity with appropriate permissions must be provided for the ALB controller to update the control plane.
\n
Application Load Balancer (ALB) Controller: The Application Gateway for Containers ALB Controller is a vital Kubernetes deployment that facilitates the seamless configuration and deployment of Application Gateway for Containers. By actively monitoring and responding to various Kubernetes Custom Resources and Resource configurations, such as Ingress, Gateway, and ApplicationLoadBalancer, the ALB Controller ensures efficient management of Application Gateway for Containers. Deployed using Helm, the ALB Controller comprises two essential pods. The first is the alb-controller pod, which takes charge of load balancing configuration for Application Gateway for Containers based on customer preferences and intent. The second is the alb-controller-bootstrap pod, responsible for effectively managing Custom Resource Definitions (CRDs) to further optimize the deployment process.
\n
Managed proxies: These proxies route traffic directly to pods within your Azure Kubernetes Service (AKS) cluster. To ensure direct addressability, the cluster and the proxies must belong to the same virtual network and be configured with Azure CNI. The Application Load Balancer (ALB) control plane creates the proxies inside the subnet referenced by the association. For this reason, the user-defined managed identity used by the Application Load Balancer (ALB) Controller need to be assigned the Network Contributor role on this subnet. The latter needs to have at least with at least a /24 IP address space.
\n
\n
\n
Features and Benefits
\n
Azure Application Gateway for Containers offers a range of features and benefits, including:
\n
\n
Load Balancing: The service efficiently distributes incoming traffic across multiple containers, ensuring optimal performance and scalability. For more information, seeLoad balancing features.
\n
Implementation of Gateway API: Application Gateway for Containers supports the Gateway API, which allows for the definition of routing rules and policies in a Kubernetes-native way. For more information, seeImplementation of Gateway API.
\n
Custom Health Probe: You can define custom health probes to monitor the health of your containers and automatically route traffic away from unhealthy instances. For more information, seeCustom health probe for Application Gateway for Containers.
\n
Session Affinity: The service provides session affinity, allowing you to maintain a consistent user experience by routing subsequent requests from the same client to the same container. For more information, seeApplication Gateway for Containers session affinity overview.
\n
TLS Policy: Application Gateway for Containers supports TLS termination, allowing you to offload the SSL/TLS encryption and decryption process to the gateway. For more information, seeApplication Gateway for Containers TLS policy overview.
\n
Header Rewrites: Application Gateway for Containers offers the capability to rewrite HTTP headers of client requests and responses from backend targets. Header Rewrites utilize theIngressExtensioncustom resource definition (CRD) of the Application Gateway for Containers. For more details, refer to the documentation on Header Rewrites forIngress APIandGateway API.
\n
URL Rewrites: Application Gateway for Containers allows you to modify the URL of a client request, including the hostname and/or path. When Application Gateway for Containers initiates the request to the backend target, it includes the newly rewritten URL. Additional information on URL Rewrites can be found in the documentation forIngress APIandGateway API.
\n
\n
\n
Advanced Load Balancing
\n
The Application Gateway for Containers offers an impressive array of traffic management features to enhance your application deployment:
\n
\n
Layer 7 HTTP/HTTPS request forwarding capabilities based on prefix/exact match criteria, including hostname, path, headers, query strings, methods, and ports (80/443).
\n
Robust support for HTTPS traffic management, including SSL termination and end-to-end SSL encryption.
\n
Seamless integration with Ingress and Gateway API for streamlined configuration and management.
\n
Flexible traffic splitting and weighted round-robin functionality to distribute traffic efficiently.
\n
Mutual Authentication (mTLS) support for establishing secure connections to backend targets.
\n
Robust health checks to ensure backends are healthy and capable of handling traffic before they are registered.
\n
Automatic retries to optimize delivery of requests and handle potential failures gracefully.
\n
TLS Policies that allow for granular control over the encryption protocols and ciphers used for secure communication.
\n
Autoscaling capabilities to dynamically adjust resources based on workload demands.
\n
Built-in resilience to handle availability zone failures and ensure continuous operation of your applications.
\n
\n
With these comprehensive features, the Application Gateway for Containers empowers you to efficiently manage and optimize your traffic flow. For more information, see For more information, seeLoad balancing features.
\n
\n
Tutorials and Samples
\n
You can use the following tutorials to begin your journey with the Application Gateway for Containers:
You can find scripts and YAML manifests for the above tutorials under thetutorialsfolder.
\n
\n
Deploy the Bicep modules
\n
You can deploy the Bicep modules in thebicepfolder using thedeploy.shBash script located in the same folder. Specify a value for the following parameters in thedeploy.shscript andmain.parameters.jsonparameters file before deploying the Bicep modules.
\n
\n
\n
prefix: specifies a prefix for all the Azure resources.
\n
authenticationType: specifies the type of authentication when accessing the Virtual Machine.sshPublicKeyis the recommended value. Allowed values:sshPublicKeyandpassword.
\n
applicationGatewayForContainersType: this parameter specifies the deployment type for the Application Gateway for Containers:\n
\n
managed: the Application Gateway for Containers resource and its child resources, association and frontends, are created and handled by the Azure Loab Balancer controller in the node resource group ofthe AKS cluster.
\n
byo: the Application Gateway for Containers resource and its child resources are created in the targert resource group. You are responsible for the provisioning and deletion of the association and frontend child resources.
\n
\n
\n
vmAdminUsername: specifies the name of the administrator account of the virtual machine.
\n
vmAdminPasswordOrKey: specifies the SSH Key or password for the virtual machine.
\n
aksClusterSshPublicKey: specifies the SSH Key or password for AKS cluster agent nodes.
\n
aadProfileAdminGroupObjectIDs: when deploying an AKS cluster with Azure AD and Azure RBAC integration, this array parameter contains the list of Azure AD group object IDs that will have the admin role of the cluster.
\n
keyVaultObjectIds: Specifies the object ID of the service principals to configure in Key Vault access policies.
\n
windowsAgentPoolEnabled: Specifies whether to create a Windows Server agent pool.
The following table contains the code from theapplicationGatewayForContainers.bicepBicep module used to deploy aApplication Gateway for Containers.
\n
\n
// Parameters\n@description('Specifies the name of the Application Gateway for Containers.')\nparam name string = 'dummy'\n\n@description('Specifies whether the Application Gateway for Containers is managed or bring your own (BYO).')\n@allowed([\n 'managed'\n 'byo'\n])\nparam type string = 'managed'\n\n@description('Specifies the workspace id of the Log Analytics used to monitor the Application Gateway for Containers.')\nparam workspaceId string\n\n@description('Specifies the location of the Application Gateway for Containers.')\nparam location string\n\n@description('Specifies the name of the existing AKS cluster.')\nparam aksClusterName string\n\n@description('Specifies the name of the AKS cluster node resource group. This needs to be passed as a parameter and cannot be calculated inside this module.')\nparam nodeResourceGroupName string\n\n@description('Specifies the name of the existing virtual network.')\nparam virtualNetworkName string\n\n@description('Specifies the name of the subnet which contains the Application Gateway for Containers.')\nparam subnetName string\n\n@description('Specifies the namespace for the Application Load Balancer Controller of the Application Gateway for Containers.')\nparam namespace string = 'azure-alb-system'\n\n@description('Specifies the name of the service account for the Application Load Balancer Controller of the Application Gateway for Containers.')\nparam serviceAccountName string = 'alb-controller-sa'\n\n@description('Specifies the resource tags for the Application Gateway for Containers.')\nparam tags object\n\n// Variables\nvar diagnosticSettingsName = 'diagnosticSettings'\nvar logCategories = [\n 'TrafficControllerAccessLog'\n]\nvar metricCategories = [\n 'AllMetrics'\n]\nvar logs = [for category in logCategories: {\n category: category\n enabled: true\n}]\nvar metrics = [for category in metricCategories: {\n category: category\n enabled: true\n}]\n\n// Resources\nresource aksCluster 'Microsoft.ContainerService/managedClusters@2024-01-02-preview' existing = {\n name: aksClusterName\n}\n\nresource virtualNetwork 'Microsoft.Network/virtualNetworks@2021-08-01' existing = {\n name: virtualNetworkName\n}\n\nresource subnet 'Microsoft.Network/virtualNetworks/subnets@2021-08-01' existing = {\n parent: virtualNetwork\n name: subnetName\n}\n\nresource readerRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {\n name: 'acdd72a7-3385-48ef-bd42-f606fba81ae7'\n scope: subscription()\n}\n\nresource networkContributorRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {\n name: '4d97b98b-1d4f-4787-a291-c67834d212e7'\n scope: subscription()\n}\n\nresource appGwForContainersConfigurationManagerRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {\n name: 'fbc52c3f-28ad-4303-a892-8a056630b8f1'\n scope: subscription()\n}\n\nresource applicationLoadBalancerManagedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2018-11-30' = {\n name: '${name}ManagedIdentity'\n location: location\n tags: tags\n}\n\n// Assign the Network Contributor role to the Application Load Balancer user-assigned managed identity with the association subnet as as scope\nresource subnetNetworkContributorRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {\n name: guid(name, applicationLoadBalancerManagedIdentity.name, networkContributorRole.id)\n scope: subnet\n properties: {\n roleDefinitionId: networkContributorRole.id\n principalId: applicationLoadBalancerManagedIdentity.properties.principalId\n principalType: 'ServicePrincipal'\n }\n}\n\n// Assign the AppGw for Containers Configuration Manager role to the Application Load Balancer user-assigned managed identity with the resource group as a scope\nresource appGwForContainersConfigurationManagerRoleAssignmenOnResourceGroup 'Microsoft.Authorization/roleAssignments@2022-04-01' = if (type == 'byo') {\n name: guid(name, applicationLoadBalancerManagedIdentity.name, appGwForContainersConfigurationManagerRole.id)\n scope: resourceGroup()\n properties: {\n roleDefinitionId: appGwForContainersConfigurationManagerRole.id\n principalId: applicationLoadBalancerManagedIdentity.properties.principalId\n principalType: 'ServicePrincipal'\n }\n}\n\n// Assign the AppGw for Containers Configuration Manager role to the Application Load Balancer user-assigned managed identity with the AKS cluster node resource group as a scope\nmodule appGwForContainersConfigurationManagerRoleAssignmenOnnodeResourceGroupName 'resourceGroupRoleAssignment.bicep' = if (type == 'managed') {\n name: guid(nodeResourceGroupName, applicationLoadBalancerManagedIdentity.name, appGwForContainersConfigurationManagerRole.id)\n scope: resourceGroup(nodeResourceGroupName)\n params: {\n principalId: applicationLoadBalancerManagedIdentity.properties.principalId\n roleName: appGwForContainersConfigurationManagerRole.name\n }\n}\n\n// Assign the Reader role the Application Load Balancer user-assigned managed identity with the AKS cluster node resource group as a scope\nmodule nodeResourceGroupReaderRoleAssignment 'resourceGroupRoleAssignment.bicep' = {\n name: guid(nodeResourceGroupName, applicationLoadBalancerManagedIdentity.name, readerRole.id)\n scope: resourceGroup(nodeResourceGroupName)\n params: {\n principalId: applicationLoadBalancerManagedIdentity.properties.principalId\n roleName: readerRole.name\n }\n}\n\n// Create federated identity for the Application Load Balancer user-assigned managed identity\nresource federatedIdentityCredentials 'Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials@2023-01-31' = if (!empty(namespace) && !empty(serviceAccountName)) {\n name: 'azure-alb-identity'\n parent: applicationLoadBalancerManagedIdentity\n properties: {\n issuer: aksCluster.properties.oidcIssuerProfile.issuerURL\n subject: 'system:serviceaccount:${namespace}:${serviceAccountName}'\n audiences: [\n 'api://AzureADTokenExchange'\n ]\n }\n}\n\nresource applicationGatewayForContainers 'Microsoft.ServiceNetworking/trafficControllers@2023-11-01' = if (type == 'byo') {\n name: name\n location: location\n tags: tags\n}\n\nresource applicationGatewayDiagnosticSettings 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = if (type == 'byo') {\n name: diagnosticSettingsName\n scope: applicationGatewayForContainers\n properties: {\n workspaceId: workspaceId\n logs: logs\n metrics: metrics\n }\n}\n\n// Outputs\noutput id string = applicationGatewayForContainers.id\noutput name string = applicationGatewayForContainers.name\noutput type string = applicationGatewayForContainers.type\noutput principalId string = applicationLoadBalancerManagedIdentity.properties.principalId\noutput clientId string = applicationLoadBalancerManagedIdentity.properties.clientId
\n
\n
The provided Bicep module performs the following steps:
\n
\n
\n
Accepts several parameters, such as thename,type,location,tags, and more.
\n
Defines variables for diagnostic settings, such asdiagnosticSettingsName,logCategories,metricCategories,logs, andmetrics.
Ceates a user-defined managed identity for the Application Load Balancer (ALB) Controller.
\n
When thetypeparameter is set tobyo, creates an Application Gateway for Containers resource in the target resource group and sets up a diagnostics settings resource to collect logs and metrics from the Application Gateway for Containers in the specified Log Analytics workspace.
\n
Assigns theNetwork Contributorrole to the Application Load Balancer user-assigned managed identity, scoped to the subnet.
\n
When thetypeparameter is set tobyo, assigns theAppGw for Containers Configuration Managerrole to the Application Load Balancer user-assigned managed identity, scoped to the resource group. This role enables ALB Controller to access and configure the Application Gateway for Containers resource.
\n
When thetypeparameter is set tomanaged, assigns theAppGw for Containers Configuration Managerrole to the Application Load Balancer user-assigned managed identity, scoped to the AKS cluster node resource group. In this case, the Application Gateway for Containers is created and managed by the ALB Controller in the AKS node resource group.
\n
Assigns theReaderrole to the Application Load Balancer user-assigned managed identity, scoped to the AKS cluster node resource group.
\n
Creates a federated identity credentials resource to establish a federated identity for the Application Load Balancer user-assigned managed identity. This is required by the ALB Controller and uses the nameazure-alb-identityfor the federated credential.
\n
Creates anapplicationGatewayForContainersresource using the Microsoft.ServiceNetworking/trafficControllers resource type to create the Application Gateway for Containers based on the provided parameters.
\n
Creates module outputs:\n
\n
id,name, andtypeof the Application Gateway for Containers.
\n
principalIdandclientIdof the ALB Controller user-defined managed identity.
\n
\n
\n
\n
When the value of thetypeparameter is set tobyo, the Bicep module creates an Application Gateway for Containers resource in the specified target resource group. Alternatively, when thetypeparameter is set tomanaged, the ALB Controller installed via Helm in the deployment script handles the creation and management of the Application Gateway for Containers in the AKS node resource group.
\n
\n
Deployment Script
\n
The followingDeployment Scriptis used to run theinstall-alb-controller-sa.shBash script stofed in a public container of a storage container. This script installs necessary dependencies, retrieves cluster credentials, checks the cluster's type, installs Helm and Helm charts, creates namespaces and service accounts, and deploys the Application Load Balancer Controller.
\n
\n
# Install kubectl\naz aks install-cli --only-show-errors\n\n# Get AKS credentials\naz aks get-credentials \\\n --admin \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --only-show-errors\n\n# Check if the cluster is private or not\nprivate=$(az aks show --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --query apiServerAccessProfile.enablePrivateCluster \\\n --output tsv)\n\n# Install Helm\ncurl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 -o get_helm.sh -s\nchmod 700 get_helm.sh\n./get_helm.sh &>/dev/null\n\n# Add Helm repos\nhelm repo add prometheus-community https://prometheus-community.github.io/helm-charts\nhelm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx\nhelm repo add jetstack https://charts.jetstack.io\n\n# Update Helm repos\nhelm repo update\n\n# initialize variables\napplicationGatewayForContainersName=''\ndiagnosticSettingName=\"DefaultDiagnosticSettings\"\n\nif [[ $private == 'true' ]]; then\n # Log whether the cluster is public or private\n echo \"$clusterName AKS cluster is private\"\n\n # Install Prometheus\n command=\"helm upgrade prometheus prometheus-community/kube-prometheus-stack \\\n --install \\\n --create-namespace \\\n --namespace prometheus \\\n --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \\\n --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false\"\n\n az aks command invoke \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --command \"$command\"\n\n # Install NGINX ingress controller using the internal load balancer\n command=\"helm upgrade nginx-ingress ingress-nginx/ingress-nginx \\\n --install \\\n --create-namespace \\\n --namespace ingress-basic \\\n --set controller.replicaCount=3 \\\n --set controller.nodeSelector.\\\"kubernetes\\.io/os\\\"=linux \\\n --set defaultBackend.nodeSelector.\\\"kubernetes\\.io/os\\\"=linux \\\n --set controller.metrics.enabled=true \\\n --set controller.metrics.serviceMonitor.enabled=true \\\n --set controller.metrics.serviceMonitor.additionalLabels.release=\\\"prometheus\\\" \\\n --set controller.service.annotations.\\\"service\\.beta\\.kubernetes\\.io/azure-load-balancer-health-probe-request-path\\\"=/healthz\"\n\n az aks command invoke \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --command \"$command\"\n\n # Install certificate manager\n command=\"helm upgrade cert-manager jetstack/cert-manager \\\n --install \\\n --create-namespace \\\n --namespace cert-manager \\\n --version v1.14.0 \\\n --set installCRDs=true \\\n --set nodeSelector.\\\"kubernetes\\.io/os\\\"=linux \\\n --set \\\"extraArgs={--feature-gates=ExperimentalGatewayAPISupport=true}\\\"\"\n\n az aks command invoke \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --command \"$command\"\n\n # Create cluster issuer\n command=\"cat <<EOF | kubectl apply -f -\napiVersion: cert-manager.io/v1\nkind: ClusterIssuer\nmetadata:\n name: letsencrypt-nginx\nspec:\n acme:\n server: https://acme-v02.api.letsencrypt.org/directory\n email: $email\n privateKeySecretRef:\n name: letsencrypt\n solvers:\n - http01:\n ingress:\n class: nginx\n podTemplate:\n spec:\n nodeSelector:\n \"kubernetes.io/os\": linux\nEOF\"\n\n az aks command invoke \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --command \"$command\"\n\n if [[ -n \"$namespace\" && \\\n -n \"$serviceAccountName\" ]]; then\n # Create workload namespace\n command=\"kubectl create namespace $namespace\"\n\n az aks command invoke \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --command \"$command\"\n\n # Create service account\n command=\"cat <<EOF | kubectl apply -f -\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n annotations:\n azure.workload.identity/client-id: $workloadManagedIdentityClientId\n azure.workload.identity/tenant-id: $tenantId\n labels:\n azure.workload.identity/use: \"true\"\n name: $serviceAccountName\n namespace: $namespace\nEOF\"\n\n az aks command invoke \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --command \"$command\"\n fi\n\n if [[ \"$applicationGatewayForContainersEnabled\" == \"true\" \\\n && -n \"$applicationGatewayForContainersManagedIdentityClientId\" \\\n && -n \"$applicationGatewayForContainersSubnetId\" ]]; then\n \n # Install the Application Load Balancer Controller\n command=\"helm upgrade alb-controller oci://mcr.microsoft.com/application-lb/charts/alb-controller \\\n --install \\\n --create-namespace \\\n --namespace $applicationGatewayForContainersNamespace \\\n --version 1.0.0 \\\n --set albController.podIdentity.clientID=$applicationGatewayForContainersManagedIdentityClientId\"\n\n az aks command invoke \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --command \"$command\"\n\n # Create workload namespace\n command=\"kubectl create namespace alb-infra\"\n\n az aks command invoke \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --command \"$command\"\n\n if [[ \"$applicationGatewayForContainersType\" == \"managed\" ]]; then\n # Define the ApplicationLoadBalancer resource, specifying the subnet ID the Application Gateway for Containers association resource should deploy into. \n # The association establishes connectivity from Application Gateway for Containers to the defined subnet (and connected networks where applicable) to \n # be able to proxy traffic to a defined backend.\n command=\"kubectl apply -f - <<EOF\napiVersion: alb.networking.azure.io/v1\nkind: ApplicationLoadBalancer\nmetadata:\n name: alb\n namespace: alb-infra\nspec:\n associations:\n - $applicationGatewayForContainersSubnetId\nEOF\"\n az aks command invoke \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --command \"$command\"\n\n if [[ -n $nodeResourceGroupName ]]; then \\\n echo -n \"Retrieving the resource id of the Application Gateway for Containers...\"\n counter=1\n while [ $counter -le 600 ]\n do\n # Retrieve the resource id of the managed Application Gateway for Containers resource\n applicationGatewayForContainersId=$(az resource list \\\n --resource-type \"Microsoft.ServiceNetworking/TrafficControllers\" \\\n --resource-group $nodeResourceGroupName \\\n --query [0].id \\\n --output tsv)\n if [[ -n $applicationGatewayForContainersId ]]; then\n echo \n break \n else\n echo -n '.'\n counter=$((counter + 1))\n sleep 1\n fi\n done\n\n if [[ -n $applicationGatewayForContainersId ]]; then\n applicationGatewayForContainersName=$(basename $applicationGatewayForContainersId)\n echo \"[$applicationGatewayForContainersId] resource id of the [$applicationGatewayForContainersName] Application Gateway for Containers successfully retrieved\"\n else\n echo \"Failed to retrieve the resource id of the Application Gateway for Containers\"\n exit -1\n fi\n\n # Check if the diagnostic setting already exists for the Application Gateway for Containers\n echo \"Checking if the [$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers actually exists...\"\n result=$(az monitor diagnostic-settings show \\\n --name $diagnosticSettingName \\\n --resource $applicationGatewayForContainersId \\\n --query name \\\n --output tsv 2>/dev/null)\n\n if [[ -z $result ]]; then\n echo \"[$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers does not exist\"\n echo \"Creating [$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers...\"\n\n # Create the diagnostic setting for the Application Gateway for Containers\n az monitor diagnostic-settings create \\\n --name $diagnosticSettingName \\\n --resource $applicationGatewayForContainersId \\\n --logs '[{\"categoryGroup\": \"allLogs\", \"enabled\": true}]' \\\n --metrics '[{\"category\": \"AllMetrics\", \"enabled\": true}]' \\\n --workspace $workspaceId \\\n --only-show-errors 1>/dev/null\n\n if [[ $? == 0 ]]; then\n echo \"[$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers successfully created\"\n else\n echo \"Failed to create [$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers\"\n exit -1\n fi\n else\n echo \"[$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers already exists\"\n fi\n fi\n fi\n fi\nelse\n # Log whether the cluster is public or private\n echo \"$clusterName AKS cluster is public\"\n\n # Install Prometheus\n echo \"Installing Prometheus...\"\n helm upgrade prometheus prometheus-community/kube-prometheus-stack \\\n --install \\\n --create-namespace \\\n --namespace prometheus \\\n --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \\\n --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false\n\n if [[ $? == 0 ]]; then\n echo \"Prometheus successfully installed\"\n else\n echo \"Failed to install Prometheus\"\n exit -1\n fi\n\n # Install NGINX ingress controller using the internal load balancer\n echo \"Installing NGINX ingress controller...\"\n helm upgrade nginx-ingress ingress-nginx/ingress-nginx \\\n --install \\\n --create-namespace \\\n --namespace ingress-basic \\\n --set controller.replicaCount=3 \\\n --set controller.nodeSelector.\"kubernetes\\.io/os\"=linux \\\n --set defaultBackend.nodeSelector.\"kubernetes\\.io/os\"=linux \\\n --set controller.metrics.enabled=true \\\n --set controller.metrics.serviceMonitor.enabled=true \\\n --set controller.metrics.serviceMonitor.additionalLabels.release=\"prometheus\" \\\n --set controller.service.annotations.\"service\\.beta\\.kubernetes\\.io/azure-load-balancer-health-probe-request-path\"=/healthz\n\n if [[ $? == 0 ]]; then\n echo \"NGINX ingress controller successfully installed\"\n else\n echo \"Failed to install NGINX ingress controller\"\n exit -1\n fi\n\n # Install certificate manager\n echo \"Installing certificate manager...\"\n helm upgrade cert-manager jetstack/cert-manager \\\n --install \\\n --create-namespace \\\n --namespace cert-manager \\\n --version v1.14.0 \\\n --set installCRDs=true \\\n --set nodeSelector.\"kubernetes\\.io/os\"=linux \\\n --set \"extraArgs={--feature-gates=ExperimentalGatewayAPISupport=true}\"\n\n if [[ $? == 0 ]]; then\n echo \"Certificate manager successfully installed\"\n else\n echo \"Failed to install certificate manager\"\n exit -1\n fi\n\n # Create cluster issuer\n echo \"Creating cluster issuer...\"\n cat <<EOF | kubectl apply -f -\napiVersion: cert-manager.io/v1\nkind: ClusterIssuer\nmetadata:\n name: letsencrypt-nginx\nspec:\n acme:\n server: https://acme-v02.api.letsencrypt.org/directory\n email: $email\n privateKeySecretRef:\n name: letsencrypt\n solvers:\n - http01:\n ingress:\n class: nginx\n podTemplate:\n spec:\n nodeSelector:\n \"kubernetes.io/os\": linux\nEOF\n\n if [[ -n \"$namespace\" && \\\n -n \"$serviceAccountName\" ]]; then\n # Create workload namespace\n result=$(kubectl get namespace -o 'jsonpath={.items[?(@.metadata.name==\"'$namespace'\")].metadata.name'})\n\n if [[ -n $result ]]; then\n echo \"$namespace namespace already exists in the cluster\"\n else\n echo \"$namespace namespace does not exist in the cluster\"\n echo \"Creating $namespace namespace in the cluster...\"\n kubectl create namespace $namespace\n fi\n\n # Create service account\n echo \"Creating $serviceAccountName service account...\"\n cat <<EOF | kubectl apply -f -\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n annotations:\n azure.workload.identity/client-id: $workloadManagedIdentityClientId\n azure.workload.identity/tenant-id: $tenantId\n labels:\n azure.workload.identity/use: \"true\"\n name: $serviceAccountName\n namespace: $namespace\nEOF\n fi\n\n if [[ \"$applicationGatewayForContainersEnabled\" == \"true\" \\\n && -n \"$applicationGatewayForContainersManagedIdentityClientId\" \\\n && -n \"$applicationGatewayForContainersSubnetId\" ]]; then\n \n # Install the Application Load Balancer\n echo \"Installing Application Load Balancer Controller in $applicationGatewayForContainersNamespace namespace using $applicationGatewayForContainersManagedIdentityClientId managed identity...\"\n helm upgrade alb-controller oci://mcr.microsoft.com/application-lb/charts/alb-controller \\\n --install \\\n --create-namespace \\\n --namespace $applicationGatewayForContainersNamespace \\\n --version 1.0.0 \\\n --set albController.namespace=$applicationGatewayForContainersNamespace \\\n --set albController.podIdentity.clientID=$applicationGatewayForContainersManagedIdentityClientId\n \n if [[ $? == 0 ]]; then\n echo \"Application Load Balancer Controller successfully installed\"\n else\n echo \"Failed to install Application Load Balancer Controller\"\n exit -1\n fi\n\n if [[ \"$applicationGatewayForContainersType\" == \"managed\" ]]; then\n # Create alb-infra namespace\n albInfraNamespace='alb-infra'\n result=$(kubectl get namespace -o 'jsonpath={.items[?(@.metadata.name==\"'$albInfraNamespace'\")].metadata.name'})\n\n if [[ -n $result ]]; then\n echo \"$albInfraNamespace namespace already exists in the cluster\"\n else\n echo \"$albInfraNamespace namespace does not exist in the cluster\"\n echo \"Creating $albInfraNamespace namespace in the cluster...\"\n kubectl create namespace $albInfraNamespace\n fi\n\n # Define the ApplicationLoadBalancer resource, specifying the subnet ID the Application Gateway for Containers association resource should deploy into. \n # The association establishes connectivity from Application Gateway for Containers to the defined subnet (and connected networks where applicable) to \n # be able to proxy traffic to a defined backend.\n echo \"Creating ApplicationLoadBalancer resource...\"\n kubectl apply -f - <<EOF\napiVersion: alb.networking.azure.io/v1\nkind: ApplicationLoadBalancer\nmetadata:\n name: alb\n namespace: alb-infra\nspec:\n associations:\n - $applicationGatewayForContainersSubnetId\nEOF\n if [[ -n $nodeResourceGroupName ]]; then \\\n echo -n \"Retrieving the resource id of the Application Gateway for Containers...\"\n counter=1\n while [ $counter -le 20 ]\n do\n # Retrieve the resource id of the managed Application Gateway for Containers resource\n applicationGatewayForContainersId=$(az resource list \\\n --resource-type \"Microsoft.ServiceNetworking/TrafficControllers\" \\\n --resource-group $nodeResourceGroupName \\\n --query [0].id \\\n --output tsv)\n if [[ -n $applicationGatewayForContainersId ]]; then\n echo \n break \n else\n echo -n '.'\n counter=$((counter + 1))\n sleep 1\n fi\n done\n\n if [[ -n $applicationGatewayForContainersId ]]; then\n applicationGatewayForContainersName=$(basename $applicationGatewayForContainersId)\n echo \"[$applicationGatewayForContainersId] resource id of the [$applicationGatewayForContainersName] Application Gateway for Containers successfully retrieved\"\n else\n echo \"Failed to retrieve the resource id of the Application Gateway for Containers\"\n exit -1\n fi\n\n # Check if the diagnostic setting already exists for the Application Gateway for Containers\n echo \"Checking if the [$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers actually exists...\"\n result=$(az monitor diagnostic-settings show \\\n --name $diagnosticSettingName \\\n --resource $applicationGatewayForContainersId \\\n --query name \\\n --output tsv 2>/dev/null)\n\n if [[ -z $result ]]; then\n echo \"[$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers does not exist\"\n echo \"Creating [$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers...\"\n\n # Create the diagnostic setting for the Application Gateway for Containers\n az monitor diagnostic-settings create \\\n --name $diagnosticSettingName \\\n --resource $applicationGatewayForContainersId \\\n --logs '[{\"categoryGroup\": \"allLogs\", \"enabled\": true}]' \\\n --metrics '[{\"category\": \"AllMetrics\", \"enabled\": true}]' \\\n --workspace $workspaceId \\\n --only-show-errors 1>/dev/null\n\n if [[ $? == 0 ]]; then\n echo \"[$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers successfully created\"\n else\n echo \"Failed to create [$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers\"\n exit -1\n fi\n else\n echo \"[$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers already exists\"\n fi\n fi\n fi\n fi\nfi\n\n# Create output as JSON file\necho '{}' |\n jq --arg x $applicationGatewayForContainersName '.applicationGatewayForContainersName=$x' |\n jq --arg x $namespace '.namespace=$x' |\n jq --arg x $serviceAccountName '.serviceAccountName=$x' |\n jq --arg x 'prometheus' '.prometheus=$x' |\n jq --arg x 'cert-manager' '.certManager=$x' |\n jq --arg x 'ingress-basic' '.nginxIngressController=$x' >$AZ_SCRIPTS_OUTPUT_PATH
\n
\n
The script performs the following steps:
\n
\n\n
Installskubectlusing the Azure CLI commandaz aks install-cli.
\n
Retrieves the AKS cluster credentials using the Azure CLI commandaz aks get-credentials.
\n
Checks whether the AKS cluster is private or public by querying theenablePrivateClusterattribute of the cluster's API server access profile using the Azure CLI commandaz aks show.
\n
InstallsHelmby downloading and executing theget_helm.shscript.
Updates the Helm repositories using thehelm repo updatecommand.
\n
Initializes variables related to the Application Gateway for Containers.
\n
The script performs the subsequent steps differently depending on whether the cluster is public or private. The script uses theaz aks command invoketo execute commands when the cluster is private.
\n
Installs Prometheus using thehelm upgrade --installcommand.
\n
Installs the certificate manager using thehelm upgrade --installcommand.
\n
If anamespaceandserviceAccountNameare provided, it creates the namespace and service account usingkubectl. This information is optional, and it can be used to create the namespace and service account for a workload.
\n
If the Application Gateway for Containers is enabled and necessary information is provided, it installs the Application Load Balancer Controller using thehelm upgrade --installcommand. The YAML manifest specifies the client id of the ALB Controller managed identity from theapplicationGatewayForContainersManagedIdentityClientIdenvironment variable and the target namespace from theapplicationGatewayForContainersNamespaceenvironment variable. For more information on the installation of the ALB Controller via Helm, seeQuickstart: Deploy Application Gateway for Containers ALB Controller.
\n
When theapplicationGatewayForContainersTypeenvironment variable is set tomanaged, creates thealb-infranamespace usingkubectland deploys theApplicationLoadBalancerresource in the newly created namespace. The YAML manifest specifies the resource id of the subnet used by the association from theapplicationGatewayForContainersSubnetIdenvironment variable.
\n
Retrieves the resource ID of the Application Gateway for Containers and checks if the diagnostic settings exist. If not, it creates the diagnostic settings usingaz monitor diagnostic-settings create.
\n
Creates an output JSON file containing the Application Gateway for Containersname, worloadnamespaceandservice account name, if any, and the namespace for Prometheus and the Certificate Manager.
\n\n
\n
Review deployed resources
\n
You can use the Azure portal to list the deployed resources in the resource group. If you chose to deploy an Application Gateway for Containers managed by the ALB Controller, you will find the resource under the node resource group of the AKS cluster.
\n
\n
\n
\n
You can also use Azure CLI to list the deployed resources in the resource group and AKS node resource group:
\n
\n
az resource list --resource-group <resource-group-name>\nnodeResourceGroupName=$(az aks show --name <aks-name> --resource-group <resource-group-name> --query \"nodeResourceGroup\" -o tsv)\naz resource list --resource-group nodeResourceGroupName
\n
\n
You can also use the following PowerShell cmdlet to list the deployed resources in the resource group and AKS node resource group:
After confirming the successful deployment, you can easily deploy your workloads and configure them to have a public endpoint through the newly created Application Gateway for Containers. To achieve this, you have two options: either aGatewayor anIngressin Kubernetes. These options allow you to expose your application to the public internet using the Application Gateway for Containers resource. The documentation provides various tutorials for both the Gateway API and the Ingress API, including:
You can find the scripts and YAML manifests for these tutorials in thetutorialsfolder. Additionally, theappfolder contains two samples:byofor a bring-your-own installation of the Application Gateway for Containers, andmanagedwhich works with Application Gateway for Containers managed by the ALB Controller.
\n
For simplicity, let's focus on themanagedsample, while leaving thebyosample for the reader to review. Let's start by reviewing the YAML manifests.
\n
\n
Deployment
\n
Thedeployment.yamlfile contains the YAML definition for the deployment, the service, and a secret that contains a temporary certificate for the Gatewy listener that will be replaced by the certificate issued byLet's Encryptvia the certificate manager. For more information on how to use the certificate manager to issue a new certificate to a Gateway using HTTP01 challenges, seeConfiguring the HTTP-01 Gateway API solver.
Thegateway.yamlcontains the definition of the Gateway used by the application. When using an Application Gateway for Containers managed by the ALB Controller, the frontend is auotmatically created for your by the ALB Controller.
The Gateway defines a certificate issuer in thecert-manager.io/issuerannotation, so we need to create an issuer. In the issuer, we define the CA root,Let's Encryptin this case, for the certificate chain to issue our certificate and the challenge type that our client would like to handle to prove our controll over the domain (in our case we will use HTTP01 challenge).
The certificate manager follows a series of steps to issue a certificate to the Gateway listener:
\n
\n\n
The gateway creates a Certificate object with a nil revision flag, initially pointing to a self-signed TLS secret. However, since the CA of the TLS secret is not valid, the Certificate realizes this and proceeds to the next step.
\n
A CertificateRequest (CR) object is created with the revision flag set to 1, indicating the need to re-issue a valid certificate. The CR contains all the necessary information to send a CSR request in PKCS #10 format to the CA.
\n
The CR object creates an Order object to monitor the request process.
\n
The Cluster Issuer registers itself in the ACME server (CA) using our public key, which is included in the CR. The CA server generates a unique token/key for our request and associates them with our public key, enabling the verification of our signature for future requests from our client.
\n
The CA server returns the unique token and key for each supported challenge to our client (Cluster Issuer), which were stored for our public key on the CA server.
\n
The cluster issuer updates the Order object with the server-supported challenges, along with their unique token and key.
\n
Based on the supported challenges and our Issuer configuration, the Order object determines to solve the HTTP01 Challenge, utilizing the parameters provided by the ACME server.
\n
A new pod is created in the default namespace to run the Challenge. This pod contains an HTTP server that serves on a specific path and expects the origin to be our domain name. If the origin does not match, a 404 error is returned.
\n\n
\n
HTTPRoute
\n
Thehttproute.yamlcontains the definition of theHTTPRouteobject used to route requests to the service:
Run the01-install-cert-manager.shscript if you need to install the certificate manager in your AKS cluster.
\n
\n
#/bin/bash\n\n# Variables\nsource ./00-variables.sh\n\n# Check if the cert-manager repository is not already added\nresult=$(helm repo list | grep $cmRepoName | awk '{print $1}')\n\nif [[ -n $result ]]; then\n echo \"[$cmRepoName] Helm repo already exists\"\nelse\n # Add the Jetstack Helm repository\n echo \"Adding [$cmRepoName] Helm repo...\"\n helm repo add $cmRepoName $cmRepoUrl\nfi\n\n# Update your local Helm chart repository cache\necho 'Updating Helm repos...'\nhelm repo update\n\n# Install cert-manager Helm chart\nresult=$(helm list -n $cmNamespace | grep $cmReleaseName | awk '{print $1}')\n\nif [[ -n $result ]]; then\n echo \"[$cmReleaseName] cert-manager already exists in the $cmNamespace namespace\"\n echo \"Upgrading [$cmReleaseName] cert-manager to the $cmNamespace namespace...\"\nelse\n # Install the cert-manager Helm chart\n echo \"Deploying [$cmReleaseName] cert-manager to the $cmNamespace namespace...\"\nfi\n\nhelm upgrade $cmReleaseName $cmRepoName/$cmChartName \\\n --install \\\n --create-namespace \\\n --namespace $cmNamespace \\\n --version $cmVersion \\\n --set installCRDs=true \\\n --set nodeSelector.\"kubernetes\\.io/os\"=linux \\\n --set \"extraArgs={--feature-gates=ExperimentalGatewayAPISupport=true}\"
\n
\n
Then run the02-create-sample.shscript to deploy the application to the specified namespace. The script makes use of theyqtool.
\n
\n
#!/bin/bash\n\n# Variables\nsource ./00-variables.sh\n\n# Check if namespace exists in the cluster\nresult=$(kubectl get namespace -o jsonpath=\"{.items[?(@.metadata.name=='$namespace')].metadata.name}\")\n\nif [[ -n $result ]]; then\n echo \"$namespace namespace already exists in the cluster\"\nelse\n echo \"$namespace namespace does not exist in the cluster\"\n echo \"creating $namespace namespace in the cluster...\"\n kubectl create namespace $namespace\nfi\n\n# Create a sample web application\nkubectl apply -n $namespace -f ./deployment.yaml\n\n# Create Gateway\ncat gateway.yaml |\n yq \"(.metadata.name)|=\"\\\"\"$gatewayName\"\\\" |\n yq \"(.metadata.namespace)|=\"\\\"\"$namespace\"\\\" |\n yq \"(.metadata.annotations.\"\\\"\"cert-manager.io/issuer\"\\\"\")|=\"\\\"\"$issuerName\"\\\" |\n yq \"(.metadata.annotations.\"\\\"\"alb.networking.azure.io/alb-name\"\\\"\")|=\"\\\"\"$applicationLoadBalancerName\"\\\" |\n yq \"(.metadata.annotations.\"\\\"\"alb.networking.azure.io/alb-namespace\"\\\"\")|=\"\\\"\"$applicationLoadBalancerNamespace\"\\\" |\n yq \"(.spec.listeners[1].hostname)|=\"\\\"\"$hostname\"\\\" |\nkubectl apply -f -\n\n# Create Issuer\ncat issuer.yaml |\n yq \"(.metadata.name)|=\"\\\"\"$issuerName\"\\\" |\n yq \"(.metadata.namespace)|=\"\\\"\"$namespace\"\\\" |\n yq \"(.spec.acme.solvers[0].http01.gatewayHTTPRoute.parentRefs[0].name)|=\"\\\"\"$gatewayName\"\\\" |\n yq \"(.spec.acme.solvers[0].http01.gatewayHTTPRoute.parentRefs[0].namespace)|=\"\\\"\"$namespace\"\\\" |\nkubectl apply -f -\n\n# Create HTTPRoute\ncat httproute.yaml |\n yq \"(.metadata.name)|=\"\\\"\"$httpRouteName\"\\\" |\n yq \"(.metadata.namespace)|=\"\\\"\"$namespace\"\\\" |\n yq \"(.spec.parentRefs[0].name)|=\"\\\"\"$gatewayName\"\\\" |\n yq \"(.spec.parentRefs[0].namespace)|=\"\\\"\"$namespace\"\\\" |\nkubectl apply -f -
\n
\n
If you delegated the management of your public DNS to Azure DNS, you can use the03-configure-dns.shscript to create a CNAME for the FQDN assigned to the frontend used by the Gateway.
\n
\n
# Variables\nsource ./00-variables.sh\n\n# Get the FQDN of the gateway\necho -n \"Retrieving the FQDN of the [$gatewayName] gateway...\"\nwhile true\ndo\n fqdn=$(kubectl get gateway $gatewayName -n $namespace -o jsonpath='{.status.addresses[0].value}')\n if [[ -n $fqdn ]]; then\n echo \n break \n else\n echo -n '.'\n sleep 1\n fi\ndone\n\nif [ -n $fqdn ]; then\n echo \"[$fqdn] FQDN successfully retrieved from the [$gatewayName] gateway\"\nelse\n echo \"Failed to retrieve the FQDN from the [$gatewayName] gateway\"\n exit\nfi\n\n# Check if an CNAME record for todolist subdomain exists in the DNS Zone\necho \"Retrieving the CNAME for the [$subdomain] subdomain from the [$dnsZoneName] DNS zone...\"\ncname=$(az network dns record-set cname list \\\n --zone-name $dnsZoneName \\\n --resource-group $dnsZoneResourceGroupName \\\n --query \"[?name=='$subdomain'].CNAMERecord.cname\" \\\n --output tsv \\\n --only-show-errors)\n\nif [[ -n $cname ]]; then\n echo \"A CNAME already exists in [$dnsZoneName] DNS zone for the [$subdomain]\"\n\n if [[ $cname == $fqdn ]]; then\n echo \"The [$cname] CNAME equals the FQDN of the [$gatewayName] gateway. No additional step is required.\"\n exit\n else\n echo \"The [$cname] CNAME is different than the [$fqdn] FQDN of the [$gatewayName] gateway\"\n fi\n\n # Delete the CNAME record\n echo \"Deleting the [$subdomain] CNAME from the [$dnsZoneName] zone...\"\n\n az network dns record-set cname delete \\\n --name $subdomain \\\n --zone-name $dnsZoneName \\\n --resource-group $dnsZoneResourceGroupName \\\n --only-show-errors \\\n --yes\n\n if [[ $? == 0 ]]; then\n echo \"[$subdomain] CNAME successfully deleted from the [$dnsZoneName] zone\"\n else\n echo \"Failed to delete the [$subdomain] CNAME from the [$dnsZoneName] zone\"\n exit\n fi\nelse\n echo \"No CNAME exists in [$dnsZoneName] DNS zone for the [$subdomain] subdomain\"\nfi\n\n# Create a CNAME record\necho \"Creating a CNAME in the [$dnsZoneName] DNS zone for the [$fqdn] FQDN of the [$gatewayName] gateway...\"\naz network dns record-set cname set-record \\\n --cname $fqdn \\\n --zone-name $dnsZoneName \\\n --resource-group $dnsZoneResourceGroupName \\\n --record-set-name $subdomain \\\n --only-show-errors 1>/dev/null\n\nif [[ $? == 0 ]]; then\n echo \"[$subdomain] CNAME successfully created in the [$dnsZoneName] DNS zone for the [$fqdn] FQDN of the [$gatewayName] gateway\"\nelse\n echo \"Failed to create a CNAME in the [$dnsZoneName] DNS zone for the [$fqdn] FQDN of the [$gatewayName] gateway\"\nfi
\n
\n
Finally, you can test the sample by running the04-test-application.shscript.
\n
\n
\n
\n
\n
\n
#!/bin/bash\n\n# Variables\nsource ./00-variables.sh\n\n# Curling this FQDN should return responses from the backend as configured in the HTTPRoute\ncurl https://$hostname\n
\n
\n
\n
\n
\n
\n
\n
You can also open the application using a web browser:
\n
\n
\n
\n
Clean up resources
\n
You can delete the resource group using the following Azure CLI command when you no longer need the resources you created. This will remove all the Azure resources.
\n
\n
az group delete --name <resource-group-name>
\n
\n
Alternatively, you can use the following PowerShell cmdlet to delete the resource group and all the Azure resources.
TheApplication Gateway for Containersis a new cutting-edge Azure service that offers load balancing and dynamic traffic management for applications running in a Kubernetes cluster. As part of Azure's Application Load Balancing portfolio, this innovative product provides an enhanced experience for developers and administrators. The Application Gateway for Containers represents the evolution of theApplication Gateway Ingress Controller (AGIC)and enables Azure Kubernetes Service (AKS) customers to leverage Azure's native Application Gateway load balancer. In this article, we will guide you through the process of deploying anAzure Kubernetes Service(AKS)cluster with anApplication Gateway for Containersin a fully automated fashion, using either a bring your own (BYO) or managed by ALB deployment.
API Server VNET Integrationallows you to enable network communication between the API server and the cluster nodes without requiring a private link or tunnel. AKS clusters with API Server VNET integration provide a series of advantages, for example, they can have public network access or private cluster mode enabled or disabled without redeploying the cluster. For more information, seeCreate an Azure Kubernetes Service cluster with API Server VNet Integration.
\n
Azure NAT Gatewayto manage outbound connections initiated by AKS-hosted workloads.
\n
Event-driven Autoscaling (KEDA) add-onis a single-purpose and lightweight component that strives to make application autoscaling simple and is a CNCF Incubation project.
\n
Dapr extension for Azure Kubernetes Service (AKS)allows you to installDapr, a portable, event-driven runtime that simplifies building resilient, stateless, and stateful applications that run on the cloud and edge and embrace the diversity of languages and developer frameworks. With its sidecar architecture, Dapr helps you tackle the challenges that come with building microservices and keeps your code platform agnostic.
Vertical Pod Autoscalingallows you to automatically sets resource requests and limits on containers per workload based on past usage. VPA makes certain pods are scheduled onto nodes that have the required CPU and memory resources. For more information, seeKubernetes Vertical Pod Autoscaling.
Image Cleanerto clean up stale images on your Azure Kubernetes Service cluster.
\n
Azure Kubernetes Service (AKS) Network Observabilityis an important part of maintaining a healthy and performant Kubernetes cluster. By collecting and analyzing data about network traffic, you can gain insights into how your cluster is operating and identify potential problems before they cause outages or performance degradation.
SystemSubnet: this subnet is used for the agent nodes of thesystemnode pool.
\n
UserSubnet: this subnet is used for the agent nodes of theusernode pool.
\n
PodSubnet: this subnet is used to allocate private IP addresses to pods dynamically.
\n
ApiServerSubnet: API Server VNET Integration projects the API server endpoint directly into this delegated subnet in the virtual network where the AKS cluster is deployed.
\n
AzureBastionSubnet: a subnet for the Azure Bastion Host.
\n
VmSubnet: a subnet for a jump-box virtual machine used to connect to the (private) AKS cluster and for the private endpoints.
\n
AppGwForConSubnet: this subnet contains the proxies created by the Application Load Balancer control plane to handle and distribute the ingress traffic to the AKS-hosted pods.
\n
\n
\n
Microsoft.ServiceNetworking/trafficControllers: anApplication Gateway for Containersused as a service proxy to handle load balancing, routing, and TLS termination for AKS-hosted workloads. There are two deployment strategies for management of Application Gateway for Containers. You can decide specify the deployment strategy using theapplicationGatewayForContainersTypeparameter in themain.bicepmodule:\n
\n
Bring your own (BYO) deployment: If you choose this strategy, the Bicep module creates the Application Gateway for Containers resource in the target deployment resource group. In this case, you are responsible to createAssociationandFrontendchild resources for the Application Gateway for Containers using the Azure Portal, Bicep, Azure CLI, Terraform, or Azure REST API. Every time you want to create a newGatewayor anIngressobject in your Azure Kubernetes Service (AKS) cluster, it's your responsibility to provision aFrontendchild resource for the Application Gateway for Containers upfront and reference it in the annotations in the Gateway or Ingress object. You are also responsible for deleting any Frontend child resource after deleting a Gateway or Ingress object in Kubernetes.
\n
Managed by ALB Controller: In this deployment strategy Azure Load Balancer (ALB) Controller deployed in AKS using an Helm chart by the deployment script is responsible for the lifecycle of the Application Gateway for Containers resource and its sub resources. The ALB Controller creates Application Gateway for Containers resource in the AKS node resource group when anApplicationLoadBalancerKubernetes object is defined on the cluster. Every time you want to create a newGatewayor anIngressobject which references theApplicationLoadBalancerKubernetes object in the annotations, the ALB Controller provisions a new Frontend resource and manage its lifecycle based on the lifecycle of the Gateway or Ingress object.
Asystemnode pool in a dedicated subnet. The default node pool hosts only critical system pods and services. The worker nodes have node taint which prevents application pods from beings scheduled on this node pool.
\n
Ausernode pool hosting user workloads and artifacts in a dedicated subnet.
\n
Awindowsnode pool hosting Windows Server containers. This node pool is optionally created when the value of thewindowsAgentPoolEnabledequalstrue
Microsoft.Compute/virtualMachines: Bicep modules can optionally create a jump-box virtual machine to manage the private AKS cluster.
\n
Microsoft.Network/bastionHosts: a separate Azure Bastion is deployed in the AKS cluster virtual network to provide SSH connectivity to both agent nodes and virtual machines.
\n
Microsoft.Network/natGateways: a bring-your-own (BYO)Azure NAT Gatewayto manage outbound connections initiated by AKS-hosted workloads. The NAT Gateway is associated to theSystemSubnet,UserSubnet, andPodSubnetsubnets. TheoutboundTypeproperty of the cluster is set touserAssignedNatGatewayto specify that a BYO NAT Gateway is used for outbound connections. NOTE: you can update theoutboundTypeafter cluster creation and this will deploy or remove resources as required to put the cluster into the new egress configuration. For more information, seeUpdating outboundType after cluster creation.
\n
Microsoft.Storage/storageAccounts: this storage account is used to store the boot diagnostics logs of both the service provider and service consumer virtual machines. Boot Diagnostics is a debugging feature that allows you to view console output and screenshots to diagnose virtual machine status.
\n
Microsoft.ContainerRegistry/registries: an Azure Container Registry (ACR) to build, store, and manage container images and artifacts in a private registry for all container deployments.
Microsoft.Monitor/accounts: AnAzure Monitor workspaceis a unique environment for data collected byAzure Monitor. Each workspace has its own data repository, configuration, and permissions. Log Analytics workspaces contain logs and metrics data from multiple Azure resources, whereas Azure Monitor workspaces currently contain only metrics related toPrometheus. Azure Monitor managed service for Prometheus allows you to collect and analyze metrics at scale using a Prometheus-compatible monitoring solution, based on thePrometheus. This fully managed service allows you to use thePrometheus query language (PromQL)to analyze and alert on the performance of monitored infrastructure and workloads without having to operate the underlying infrastructure. The primary method for visualizing Prometheus metrics isAzure Managed Grafana. You can connect yourAzure Monitor workspaceto anAzure Managed Grafanato visualize Prometheus metrics using a set of built-in and custom Grafana dashboards.
The Bicep modules provide the flexibility to deploy the following Azure resources based on your requirements selectively:
\n
\n
Microsoft.CognitiveServices/accounts: anAzure OpenAI Servicewith aGPT-3.5model used by an AI application like a chatbot. Azure OpenAI Service gives customers advanced language AI with OpenAI GPT-4, GPT-3, Codex, and DALL-E models with Azure's security and enterprise promise. Azure OpenAI co-develops the APIs with OpenAI, ensuring compatibility and a smooth transition from one to the other.
NOTE You can find thearchitecture.vsdxfile used for the diagram under thevisiofolder.
\n
\n
\n
What is Bicep?
\n
Bicepis a domain-specific language (DSL) that uses a declarative syntax to deploy Azure resources. It provides concise syntax, reliable type safety, and support for code reuse. Bicep offers the best authoring experience for your infrastructure-as-code solutions in Azure.
\n
\n
What is Gateway API?
\n
The Ingress resources Kubernetes objects have evolved into the more comprehensive and powerful Kubernetes Gateway API. Ingress Controller and Gateway API are both Kubernetes objects used for managing traffic routing and load balancing. While Ingress Controller served as entry points for external traffic, they had limitations in terms of flexibility and extensibility. The Kubernetes Gateway API emerged as a solution to address these limitations. Designed to be generic, expressive, extensible, and role-oriented, the Gateway API is a modern set of APIs for defining L4 and L7 routing rules in Kubernetes.
\n
\n
\n
\n
Gateway API offers superior functionality compared to Ingress Controllers as it separates listeners and routes into separate Kubernetes objects, Gateway and HTTPRoute. This separation allows different individuals with distinct roles and permissions to deploy them in separate namespaces. Additionally, Gateway API provides advanced traffic management capabilities including layer 7 HTTP/HTTPS request forwarding based on criteria such as hostname, path, headers, query string, methods, and ports. It also offers SSL termination and TLS policies for secure traffic management. These features grant better control and customization of traffic routing. The design of the Gateway API was driven by the following design goals to address and resolve issues and limitations in ingress controllers:
\n
\n
\n
Role-oriented: The Gateway API comprises API resources that model organizational roles involved in using and configuring Kubernetes service networking.
\n
Portable: Similar to Ingress, the Gateway API is designed to be a portable specification supported by multiple implementations.
\n
Expressive: The Gateway API resources support core functionality such as header-based matching, traffic weighting, and other capabilities that were previously only possible through custom annotations in Ingress.
\n
Extensible: The Gateway API allows for the linking of custom resources at different layers of the API, enabling granular customization within the API structure.
\n
\n
Additional notable capabilities of the Gateway API include:
\n
\n
GatewayClasses: Formalizes types of load-balancing implementations, making it easier for users to understand available capabilities through the Kubernetes resource model.
\n
Shared Gateways and cross-Namespace support: Allows multiple Route resources to attach to the same Gateway, enabling load balancer and VIP sharing among teams and across Namespaces without direct coordination.
\n
Typed Routes and typed backends: The Gateway API supports typed Route resources and different types of backends, providing flexibility in supporting various protocols (HTTP, gRPC) and backend targets (Kubernetes Services, storage buckets, functions).
\n
ExperimentalService mesh supportwith the GAMMA initiative: The Gateway API enables the association of routing resources with Service resources, allowing the configuration of service meshes and ingress controllers.
Ingress Controllers are a straightforward option for setting up and are well-suited for smaller and less complex Kubernetes deployments that prioritize easy configuration.
\n
If you currently have Ingress controllers configured in your Kubernetes cluster and they meet your requirements effectively, there may not be an immediate necessity to transition to the Kubernetes Gateway API.
\n
\n
Gateway APIis the recommended option in the following situations:
\n
\n
When dealing with complex routing configurations, traffic splitting, and advanced traffic management strategies, the flexibility provided by Kubernetes Gateway API's Route resources is essential.
\n
In cases where networking requirements call for custom solutions or the integration of third-party plugins, the Kubernetes Gateway API's CRD-based approach offers enhanced extensibility.
\n
\n
What is Application Gateway for Containers?
\n
TheApplication Gateway for Containersis a new cutting-edge Azure service that offers load balancing and dynamic traffic management for applications running in a Kubernetes cluster. As part of Azure's Application Load Balancing portfolio, this innovative product provides an enhanced experience for developers and administrators. The Application Gateway for Containers represents the evolution of theApplication Gateway Ingress Controller (AGIC)and enables Azure Kubernetes Service (AKS) customers to leverage Azure's native Application Gateway load balancer. Azure Application Gateway for Containers enables you to host multiple web applications on the same port, utilizing unique backend services. This allows for efficient multi-site hosting and simplifies the management of your containerized applications. The Application Gateway for Containers fully supports both theGateway APIandIngress APIKubernetes objects for traffic load balancing. For more information, see:
Azure Application Gateway for Containers supports two main deployment strategies:
\n
\n
\n
Bring your own (BYO) deployment: If you choose this strategy, the Bicep module creates the Application Gateway for Containers resource in the target deployment resource group. In this case, you are responsible to createAssociationandFrontendchild resources for the Application Gateway for Containers using the Azure Portal, Bicep, Azure CLI, Terraform, or Azure REST API. Every time you want to create a newGatewayor anIngressobject in your Azure Kubernetes Service (AKS) cluster, it's your responsibility to provision aFrontendchild resource for the Application Gateway for Containers upfront and reference it in the annotations in the Gateway or Ingress object. After deleting a Gateway or Ingress object in Kubernetes, you are also responsible for deleting any Frontend child resource.
\n
Managed by the Application Load Balancer (ALB) Controller: In this deployment strategy Azure Load Balancer (ALB) Controller deployed in AKS using an Helm chart by the deployment script is responsible for the lifecycle of the Application Gateway for Containers resource and its sub-resources. The ALB Controller creates an Application Gateway for Containers resource in the AKS node resource group when anApplicationLoadBalancerKubernetes object is defined on the cluster. Every time you want to create a newGatewayor anIngressobject which references theApplicationLoadBalancerKubernetes object in the annotations, the ALB Controller provisions a new Frontend resource and manage its lifecycle based on the lifecycle of the Gateway or Ingress object.
\n
\n
\n
Application Gateway for Containers Components
\n
The components of Azure Application Gateway for Containers include:
\n
\n
\n
Core Components: Application Gateway for Containers is a parent Azure resource that manages the control plane, which handles the configuration and orchestration of proxies based on customer requirements. It serves as the parent resource for two important child resources: associations and frontends. These child resources are unique to each Application Gateway for Containers and cannot be shared with other instances of Application Gateway for Containers.
\n
Frontend: An Application Gateway for Containers frontend is a sub-resource of the parent Application Gateway for Containers in Azure. It acts as the entry point for client traffic directed towards a specific Application Gateway for Containers. Each frontend is unique and cannot be associated with multiple Application Gateway for Containers. It provides a unique fully qualified domain name (FQDN) that can be assigned to a customer's CNAME record. Currently, private IP addresses are not supported for frontends. Additionally, a single Application Gateway for Containers has the ability to support multiple frontends.
\n
Association: An Application Gateway for Containers association is a connection point into a virtual network and is a child resource of the Application Gateway for Containers. Application Gateway for Containers is designed to allow for multiple associations, but currently only one association is allowed. During the creation of an association, the necessary data plane is provisioned and connected to a subnet within the defined virtual network. Each association should have at least 256 available addresses in the subnet. If multiple Application Gateway for Containers are provisioned and each contains one association, the required number of available addresses should be n*256. It is important that all association resources match the same region as the parent Application Gateway for Containers resource. The subnet referenced by the association will contain the proxy components used to handle the ingress traffic to the Azure Kubernetes Service (AKS) cluster.
\n
Managed identity: A user-defined managed identity with appropriate permissions must be provided for the ALB controller to update the control plane.
\n
Application Load Balancer (ALB) Controller: The Application Gateway for Containers ALB Controller is a vital Kubernetes deployment that facilitates the seamless configuration and deployment of Application Gateway for Containers. By actively monitoring and responding to various Kubernetes Custom Resources and Resource configurations, such as Ingress, Gateway, and ApplicationLoadBalancer, the ALB Controller ensures efficient management of Application Gateway for Containers. Deployed using Helm, the ALB Controller comprises two essential pods. The first is the alb-controller pod, which takes charge of load balancing configuration for Application Gateway for Containers based on customer preferences and intent. The second is the alb-controller-bootstrap pod, responsible for effectively managing Custom Resource Definitions (CRDs) to further optimize the deployment process.
\n
Managed proxies: These proxies route traffic directly to pods within your Azure Kubernetes Service (AKS) cluster. To ensure direct addressability, the cluster and the proxies must belong to the same virtual network and be configured with Azure CNI. The Application Load Balancer (ALB) control plane creates the proxies inside the subnet referenced by the association. For this reason, the user-defined managed identity used by the Application Load Balancer (ALB) Controller need to be assigned the Network Contributor role on this subnet. The latter needs to have at least with at least a /24 IP address space.
\n
\n
\n
Features and Benefits
\n
Azure Application Gateway for Containers offers a range of features and benefits, including:
\n
\n
Load Balancing: The service efficiently distributes incoming traffic across multiple containers, ensuring optimal performance and scalability. For more information, seeLoad balancing features.
\n
Implementation of Gateway API: Application Gateway for Containers supports the Gateway API, which allows for the definition of routing rules and policies in a Kubernetes-native way. For more information, seeImplementation of Gateway API.
\n
Custom Health Probe: You can define custom health probes to monitor the health of your containers and automatically route traffic away from unhealthy instances. For more information, seeCustom health probe for Application Gateway for Containers.
\n
Session Affinity: The service provides session affinity, allowing you to maintain a consistent user experience by routing subsequent requests from the same client to the same container. For more information, seeApplication Gateway for Containers session affinity overview.
\n
TLS Policy: Application Gateway for Containers supports TLS termination, allowing you to offload the SSL/TLS encryption and decryption process to the gateway. For more information, seeApplication Gateway for Containers TLS policy overview.
\n
Header Rewrites: Application Gateway for Containers offers the capability to rewrite HTTP headers of client requests and responses from backend targets. Header Rewrites utilize theIngressExtensioncustom resource definition (CRD) of the Application Gateway for Containers. For more details, refer to the documentation on Header Rewrites forIngress APIandGateway API.
\n
URL Rewrites: Application Gateway for Containers allows you to modify the URL of a client request, including the hostname and/or path. When Application Gateway for Containers initiates the request to the backend target, it includes the newly rewritten URL. Additional information on URL Rewrites can be found in the documentation forIngress APIandGateway API.
\n
\n
\n
Advanced Load Balancing
\n
The Application Gateway for Containers offers an impressive array of traffic management features to enhance your application deployment:
\n
\n
Layer 7 HTTP/HTTPS request forwarding capabilities based on prefix/exact match criteria, including hostname, path, headers, query strings, methods, and ports (80/443).
\n
Robust support for HTTPS traffic management, including SSL termination and end-to-end SSL encryption.
\n
Seamless integration with Ingress and Gateway API for streamlined configuration and management.
\n
Flexible traffic splitting and weighted round-robin functionality to distribute traffic efficiently.
\n
Mutual Authentication (mTLS) support for establishing secure connections to backend targets.
\n
Robust health checks to ensure backends are healthy and capable of handling traffic before they are registered.
\n
Automatic retries to optimize delivery of requests and handle potential failures gracefully.
\n
TLS Policies that allow for granular control over the encryption protocols and ciphers used for secure communication.
\n
Autoscaling capabilities to dynamically adjust resources based on workload demands.
\n
Built-in resilience to handle availability zone failures and ensure continuous operation of your applications.
\n
\n
With these comprehensive features, the Application Gateway for Containers empowers you to efficiently manage and optimize your traffic flow. For more information, see For more information, seeLoad balancing features.
\n
\n
Tutorials and Samples
\n
You can use the following tutorials to begin your journey with the Application Gateway for Containers:
You can find scripts and YAML manifests for the above tutorials under thetutorialsfolder.
\n
\n
Deploy the Bicep modules
\n
You can deploy the Bicep modules in thebicepfolder using thedeploy.shBash script located in the same folder. Specify a value for the following parameters in thedeploy.shscript andmain.parameters.jsonparameters file before deploying the Bicep modules.
\n
\n
\n
prefix: specifies a prefix for all the Azure resources.
\n
authenticationType: specifies the type of authentication when accessing the Virtual Machine.sshPublicKeyis the recommended value. Allowed values:sshPublicKeyandpassword.
\n
applicationGatewayForContainersType: this parameter specifies the deployment type for the Application Gateway for Containers:\n
\n
managed: the Application Gateway for Containers resource and its child resources, association and frontends, are created and handled by the Azure Loab Balancer controller in the node resource group ofthe AKS cluster.
\n
byo: the Application Gateway for Containers resource and its child resources are created in the targert resource group. You are responsible for the provisioning and deletion of the association and frontend child resources.
\n
\n
\n
vmAdminUsername: specifies the name of the administrator account of the virtual machine.
\n
vmAdminPasswordOrKey: specifies the SSH Key or password for the virtual machine.
\n
aksClusterSshPublicKey: specifies the SSH Key or password for AKS cluster agent nodes.
\n
aadProfileAdminGroupObjectIDs: when deploying an AKS cluster with Azure AD and Azure RBAC integration, this array parameter contains the list of Azure AD group object IDs that will have the admin role of the cluster.
\n
keyVaultObjectIds: Specifies the object ID of the service principals to configure in Key Vault access policies.
\n
windowsAgentPoolEnabled: Specifies whether to create a Windows Server agent pool.
The following table contains the code from theapplicationGatewayForContainers.bicepBicep module used to deploy aApplication Gateway for Containers.
\n
\n
// Parameters\n@description('Specifies the name of the Application Gateway for Containers.')\nparam name string = 'dummy'\n\n@description('Specifies whether the Application Gateway for Containers is managed or bring your own (BYO).')\n@allowed([\n 'managed'\n 'byo'\n])\nparam type string = 'managed'\n\n@description('Specifies the workspace id of the Log Analytics used to monitor the Application Gateway for Containers.')\nparam workspaceId string\n\n@description('Specifies the location of the Application Gateway for Containers.')\nparam location string\n\n@description('Specifies the name of the existing AKS cluster.')\nparam aksClusterName string\n\n@description('Specifies the name of the AKS cluster node resource group. This needs to be passed as a parameter and cannot be calculated inside this module.')\nparam nodeResourceGroupName string\n\n@description('Specifies the name of the existing virtual network.')\nparam virtualNetworkName string\n\n@description('Specifies the name of the subnet which contains the Application Gateway for Containers.')\nparam subnetName string\n\n@description('Specifies the namespace for the Application Load Balancer Controller of the Application Gateway for Containers.')\nparam namespace string = 'azure-alb-system'\n\n@description('Specifies the name of the service account for the Application Load Balancer Controller of the Application Gateway for Containers.')\nparam serviceAccountName string = 'alb-controller-sa'\n\n@description('Specifies the resource tags for the Application Gateway for Containers.')\nparam tags object\n\n// Variables\nvar diagnosticSettingsName = 'diagnosticSettings'\nvar logCategories = [\n 'TrafficControllerAccessLog'\n]\nvar metricCategories = [\n 'AllMetrics'\n]\nvar logs = [for category in logCategories: {\n category: category\n enabled: true\n}]\nvar metrics = [for category in metricCategories: {\n category: category\n enabled: true\n}]\n\n// Resources\nresource aksCluster 'Microsoft.ContainerService/managedClusters@2024-01-02-preview' existing = {\n name: aksClusterName\n}\n\nresource virtualNetwork 'Microsoft.Network/virtualNetworks@2021-08-01' existing = {\n name: virtualNetworkName\n}\n\nresource subnet 'Microsoft.Network/virtualNetworks/subnets@2021-08-01' existing = {\n parent: virtualNetwork\n name: subnetName\n}\n\nresource readerRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {\n name: 'acdd72a7-3385-48ef-bd42-f606fba81ae7'\n scope: subscription()\n}\n\nresource networkContributorRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {\n name: '4d97b98b-1d4f-4787-a291-c67834d212e7'\n scope: subscription()\n}\n\nresource appGwForContainersConfigurationManagerRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {\n name: 'fbc52c3f-28ad-4303-a892-8a056630b8f1'\n scope: subscription()\n}\n\nresource applicationLoadBalancerManagedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2018-11-30' = {\n name: '${name}ManagedIdentity'\n location: location\n tags: tags\n}\n\n// Assign the Network Contributor role to the Application Load Balancer user-assigned managed identity with the association subnet as as scope\nresource subnetNetworkContributorRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {\n name: guid(name, applicationLoadBalancerManagedIdentity.name, networkContributorRole.id)\n scope: subnet\n properties: {\n roleDefinitionId: networkContributorRole.id\n principalId: applicationLoadBalancerManagedIdentity.properties.principalId\n principalType: 'ServicePrincipal'\n }\n}\n\n// Assign the AppGw for Containers Configuration Manager role to the Application Load Balancer user-assigned managed identity with the resource group as a scope\nresource appGwForContainersConfigurationManagerRoleAssignmenOnResourceGroup 'Microsoft.Authorization/roleAssignments@2022-04-01' = if (type == 'byo') {\n name: guid(name, applicationLoadBalancerManagedIdentity.name, appGwForContainersConfigurationManagerRole.id)\n scope: resourceGroup()\n properties: {\n roleDefinitionId: appGwForContainersConfigurationManagerRole.id\n principalId: applicationLoadBalancerManagedIdentity.properties.principalId\n principalType: 'ServicePrincipal'\n }\n}\n\n// Assign the AppGw for Containers Configuration Manager role to the Application Load Balancer user-assigned managed identity with the AKS cluster node resource group as a scope\nmodule appGwForContainersConfigurationManagerRoleAssignmenOnnodeResourceGroupName 'resourceGroupRoleAssignment.bicep' = if (type == 'managed') {\n name: guid(nodeResourceGroupName, applicationLoadBalancerManagedIdentity.name, appGwForContainersConfigurationManagerRole.id)\n scope: resourceGroup(nodeResourceGroupName)\n params: {\n principalId: applicationLoadBalancerManagedIdentity.properties.principalId\n roleName: appGwForContainersConfigurationManagerRole.name\n }\n}\n\n// Assign the Reader role the Application Load Balancer user-assigned managed identity with the AKS cluster node resource group as a scope\nmodule nodeResourceGroupReaderRoleAssignment 'resourceGroupRoleAssignment.bicep' = {\n name: guid(nodeResourceGroupName, applicationLoadBalancerManagedIdentity.name, readerRole.id)\n scope: resourceGroup(nodeResourceGroupName)\n params: {\n principalId: applicationLoadBalancerManagedIdentity.properties.principalId\n roleName: readerRole.name\n }\n}\n\n// Create federated identity for the Application Load Balancer user-assigned managed identity\nresource federatedIdentityCredentials 'Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials@2023-01-31' = if (!empty(namespace) && !empty(serviceAccountName)) {\n name: 'azure-alb-identity'\n parent: applicationLoadBalancerManagedIdentity\n properties: {\n issuer: aksCluster.properties.oidcIssuerProfile.issuerURL\n subject: 'system:serviceaccount:${namespace}:${serviceAccountName}'\n audiences: [\n 'api://AzureADTokenExchange'\n ]\n }\n}\n\nresource applicationGatewayForContainers 'Microsoft.ServiceNetworking/trafficControllers@2023-11-01' = if (type == 'byo') {\n name: name\n location: location\n tags: tags\n}\n\nresource applicationGatewayDiagnosticSettings 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = if (type == 'byo') {\n name: diagnosticSettingsName\n scope: applicationGatewayForContainers\n properties: {\n workspaceId: workspaceId\n logs: logs\n metrics: metrics\n }\n}\n\n// Outputs\noutput id string = applicationGatewayForContainers.id\noutput name string = applicationGatewayForContainers.name\noutput type string = applicationGatewayForContainers.type\noutput principalId string = applicationLoadBalancerManagedIdentity.properties.principalId\noutput clientId string = applicationLoadBalancerManagedIdentity.properties.clientId
\n
\n
The provided Bicep module performs the following steps:
\n
\n
\n
Accepts several parameters, such as thename,type,location,tags, and more.
\n
Defines variables for diagnostic settings, such asdiagnosticSettingsName,logCategories,metricCategories,logs, andmetrics.
Ceates a user-defined managed identity for the Application Load Balancer (ALB) Controller.
\n
When thetypeparameter is set tobyo, creates an Application Gateway for Containers resource in the target resource group and sets up a diagnostics settings resource to collect logs and metrics from the Application Gateway for Containers in the specified Log Analytics workspace.
\n
Assigns theNetwork Contributorrole to the Application Load Balancer user-assigned managed identity, scoped to the subnet.
\n
When thetypeparameter is set tobyo, assigns theAppGw for Containers Configuration Managerrole to the Application Load Balancer user-assigned managed identity, scoped to the resource group. This role enables ALB Controller to access and configure the Application Gateway for Containers resource.
\n
When thetypeparameter is set tomanaged, assigns theAppGw for Containers Configuration Managerrole to the Application Load Balancer user-assigned managed identity, scoped to the AKS cluster node resource group. In this case, the Application Gateway for Containers is created and managed by the ALB Controller in the AKS node resource group.
\n
Assigns theReaderrole to the Application Load Balancer user-assigned managed identity, scoped to the AKS cluster node resource group.
\n
Creates a federated identity credentials resource to establish a federated identity for the Application Load Balancer user-assigned managed identity. This is required by the ALB Controller and uses the nameazure-alb-identityfor the federated credential.
\n
Creates anapplicationGatewayForContainersresource using the Microsoft.ServiceNetworking/trafficControllers resource type to create the Application Gateway for Containers based on the provided parameters.
\n
Creates module outputs:\n
\n
id,name, andtypeof the Application Gateway for Containers.
\n
principalIdandclientIdof the ALB Controller user-defined managed identity.
\n
\n
\n
\n
When the value of thetypeparameter is set tobyo, the Bicep module creates an Application Gateway for Containers resource in the specified target resource group. Alternatively, when thetypeparameter is set tomanaged, the ALB Controller installed via Helm in the deployment script handles the creation and management of the Application Gateway for Containers in the AKS node resource group.
\n
\n
Deployment Script
\n
The followingDeployment Scriptis used to run theinstall-alb-controller-sa.shBash script stofed in a public container of a storage container. This script installs necessary dependencies, retrieves cluster credentials, checks the cluster's type, installs Helm and Helm charts, creates namespaces and service accounts, and deploys the Application Load Balancer Controller.
\n
\n
# Install kubectl\naz aks install-cli --only-show-errors\n\n# Get AKS credentials\naz aks get-credentials \\\n --admin \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --only-show-errors\n\n# Check if the cluster is private or not\nprivate=$(az aks show --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --query apiServerAccessProfile.enablePrivateCluster \\\n --output tsv)\n\n# Install Helm\ncurl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 -o get_helm.sh -s\nchmod 700 get_helm.sh\n./get_helm.sh &>/dev/null\n\n# Add Helm repos\nhelm repo add prometheus-community https://prometheus-community.github.io/helm-charts\nhelm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx\nhelm repo add jetstack https://charts.jetstack.io\n\n# Update Helm repos\nhelm repo update\n\n# initialize variables\napplicationGatewayForContainersName=''\ndiagnosticSettingName=\"DefaultDiagnosticSettings\"\n\nif [[ $private == 'true' ]]; then\n # Log whether the cluster is public or private\n echo \"$clusterName AKS cluster is private\"\n\n # Install Prometheus\n command=\"helm upgrade prometheus prometheus-community/kube-prometheus-stack \\\n --install \\\n --create-namespace \\\n --namespace prometheus \\\n --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \\\n --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false\"\n\n az aks command invoke \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --command \"$command\"\n\n # Install NGINX ingress controller using the internal load balancer\n command=\"helm upgrade nginx-ingress ingress-nginx/ingress-nginx \\\n --install \\\n --create-namespace \\\n --namespace ingress-basic \\\n --set controller.replicaCount=3 \\\n --set controller.nodeSelector.\\\"kubernetes\\.io/os\\\"=linux \\\n --set defaultBackend.nodeSelector.\\\"kubernetes\\.io/os\\\"=linux \\\n --set controller.metrics.enabled=true \\\n --set controller.metrics.serviceMonitor.enabled=true \\\n --set controller.metrics.serviceMonitor.additionalLabels.release=\\\"prometheus\\\" \\\n --set controller.service.annotations.\\\"service\\.beta\\.kubernetes\\.io/azure-load-balancer-health-probe-request-path\\\"=/healthz\"\n\n az aks command invoke \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --command \"$command\"\n\n # Install certificate manager\n command=\"helm upgrade cert-manager jetstack/cert-manager \\\n --install \\\n --create-namespace \\\n --namespace cert-manager \\\n --version v1.14.0 \\\n --set installCRDs=true \\\n --set nodeSelector.\\\"kubernetes\\.io/os\\\"=linux \\\n --set \\\"extraArgs={--feature-gates=ExperimentalGatewayAPISupport=true}\\\"\"\n\n az aks command invoke \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --command \"$command\"\n\n # Create cluster issuer\n command=\"cat <<EOF | kubectl apply -f -\napiVersion: cert-manager.io/v1\nkind: ClusterIssuer\nmetadata:\n name: letsencrypt-nginx\nspec:\n acme:\n server: https://acme-v02.api.letsencrypt.org/directory\n email: $email\n privateKeySecretRef:\n name: letsencrypt\n solvers:\n - http01:\n ingress:\n class: nginx\n podTemplate:\n spec:\n nodeSelector:\n \"kubernetes.io/os\": linux\nEOF\"\n\n az aks command invoke \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --command \"$command\"\n\n if [[ -n \"$namespace\" && \\\n -n \"$serviceAccountName\" ]]; then\n # Create workload namespace\n command=\"kubectl create namespace $namespace\"\n\n az aks command invoke \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --command \"$command\"\n\n # Create service account\n command=\"cat <<EOF | kubectl apply -f -\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n annotations:\n azure.workload.identity/client-id: $workloadManagedIdentityClientId\n azure.workload.identity/tenant-id: $tenantId\n labels:\n azure.workload.identity/use: \"true\"\n name: $serviceAccountName\n namespace: $namespace\nEOF\"\n\n az aks command invoke \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --command \"$command\"\n fi\n\n if [[ \"$applicationGatewayForContainersEnabled\" == \"true\" \\\n && -n \"$applicationGatewayForContainersManagedIdentityClientId\" \\\n && -n \"$applicationGatewayForContainersSubnetId\" ]]; then\n \n # Install the Application Load Balancer Controller\n command=\"helm upgrade alb-controller oci://mcr.microsoft.com/application-lb/charts/alb-controller \\\n --install \\\n --create-namespace \\\n --namespace $applicationGatewayForContainersNamespace \\\n --version 1.0.0 \\\n --set albController.podIdentity.clientID=$applicationGatewayForContainersManagedIdentityClientId\"\n\n az aks command invoke \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --command \"$command\"\n\n # Create workload namespace\n command=\"kubectl create namespace alb-infra\"\n\n az aks command invoke \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --command \"$command\"\n\n if [[ \"$applicationGatewayForContainersType\" == \"managed\" ]]; then\n # Define the ApplicationLoadBalancer resource, specifying the subnet ID the Application Gateway for Containers association resource should deploy into. \n # The association establishes connectivity from Application Gateway for Containers to the defined subnet (and connected networks where applicable) to \n # be able to proxy traffic to a defined backend.\n command=\"kubectl apply -f - <<EOF\napiVersion: alb.networking.azure.io/v1\nkind: ApplicationLoadBalancer\nmetadata:\n name: alb\n namespace: alb-infra\nspec:\n associations:\n - $applicationGatewayForContainersSubnetId\nEOF\"\n az aks command invoke \\\n --name $clusterName \\\n --resource-group $resourceGroupName \\\n --subscription $subscriptionId \\\n --command \"$command\"\n\n if [[ -n $nodeResourceGroupName ]]; then \\\n echo -n \"Retrieving the resource id of the Application Gateway for Containers...\"\n counter=1\n while [ $counter -le 600 ]\n do\n # Retrieve the resource id of the managed Application Gateway for Containers resource\n applicationGatewayForContainersId=$(az resource list \\\n --resource-type \"Microsoft.ServiceNetworking/TrafficControllers\" \\\n --resource-group $nodeResourceGroupName \\\n --query [0].id \\\n --output tsv)\n if [[ -n $applicationGatewayForContainersId ]]; then\n echo \n break \n else\n echo -n '.'\n counter=$((counter + 1))\n sleep 1\n fi\n done\n\n if [[ -n $applicationGatewayForContainersId ]]; then\n applicationGatewayForContainersName=$(basename $applicationGatewayForContainersId)\n echo \"[$applicationGatewayForContainersId] resource id of the [$applicationGatewayForContainersName] Application Gateway for Containers successfully retrieved\"\n else\n echo \"Failed to retrieve the resource id of the Application Gateway for Containers\"\n exit -1\n fi\n\n # Check if the diagnostic setting already exists for the Application Gateway for Containers\n echo \"Checking if the [$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers actually exists...\"\n result=$(az monitor diagnostic-settings show \\\n --name $diagnosticSettingName \\\n --resource $applicationGatewayForContainersId \\\n --query name \\\n --output tsv 2>/dev/null)\n\n if [[ -z $result ]]; then\n echo \"[$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers does not exist\"\n echo \"Creating [$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers...\"\n\n # Create the diagnostic setting for the Application Gateway for Containers\n az monitor diagnostic-settings create \\\n --name $diagnosticSettingName \\\n --resource $applicationGatewayForContainersId \\\n --logs '[{\"categoryGroup\": \"allLogs\", \"enabled\": true}]' \\\n --metrics '[{\"category\": \"AllMetrics\", \"enabled\": true}]' \\\n --workspace $workspaceId \\\n --only-show-errors 1>/dev/null\n\n if [[ $? == 0 ]]; then\n echo \"[$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers successfully created\"\n else\n echo \"Failed to create [$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers\"\n exit -1\n fi\n else\n echo \"[$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers already exists\"\n fi\n fi\n fi\n fi\nelse\n # Log whether the cluster is public or private\n echo \"$clusterName AKS cluster is public\"\n\n # Install Prometheus\n echo \"Installing Prometheus...\"\n helm upgrade prometheus prometheus-community/kube-prometheus-stack \\\n --install \\\n --create-namespace \\\n --namespace prometheus \\\n --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \\\n --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false\n\n if [[ $? == 0 ]]; then\n echo \"Prometheus successfully installed\"\n else\n echo \"Failed to install Prometheus\"\n exit -1\n fi\n\n # Install NGINX ingress controller using the internal load balancer\n echo \"Installing NGINX ingress controller...\"\n helm upgrade nginx-ingress ingress-nginx/ingress-nginx \\\n --install \\\n --create-namespace \\\n --namespace ingress-basic \\\n --set controller.replicaCount=3 \\\n --set controller.nodeSelector.\"kubernetes\\.io/os\"=linux \\\n --set defaultBackend.nodeSelector.\"kubernetes\\.io/os\"=linux \\\n --set controller.metrics.enabled=true \\\n --set controller.metrics.serviceMonitor.enabled=true \\\n --set controller.metrics.serviceMonitor.additionalLabels.release=\"prometheus\" \\\n --set controller.service.annotations.\"service\\.beta\\.kubernetes\\.io/azure-load-balancer-health-probe-request-path\"=/healthz\n\n if [[ $? == 0 ]]; then\n echo \"NGINX ingress controller successfully installed\"\n else\n echo \"Failed to install NGINX ingress controller\"\n exit -1\n fi\n\n # Install certificate manager\n echo \"Installing certificate manager...\"\n helm upgrade cert-manager jetstack/cert-manager \\\n --install \\\n --create-namespace \\\n --namespace cert-manager \\\n --version v1.14.0 \\\n --set installCRDs=true \\\n --set nodeSelector.\"kubernetes\\.io/os\"=linux \\\n --set \"extraArgs={--feature-gates=ExperimentalGatewayAPISupport=true}\"\n\n if [[ $? == 0 ]]; then\n echo \"Certificate manager successfully installed\"\n else\n echo \"Failed to install certificate manager\"\n exit -1\n fi\n\n # Create cluster issuer\n echo \"Creating cluster issuer...\"\n cat <<EOF | kubectl apply -f -\napiVersion: cert-manager.io/v1\nkind: ClusterIssuer\nmetadata:\n name: letsencrypt-nginx\nspec:\n acme:\n server: https://acme-v02.api.letsencrypt.org/directory\n email: $email\n privateKeySecretRef:\n name: letsencrypt\n solvers:\n - http01:\n ingress:\n class: nginx\n podTemplate:\n spec:\n nodeSelector:\n \"kubernetes.io/os\": linux\nEOF\n\n if [[ -n \"$namespace\" && \\\n -n \"$serviceAccountName\" ]]; then\n # Create workload namespace\n result=$(kubectl get namespace -o 'jsonpath={.items[?(@.metadata.name==\"'$namespace'\")].metadata.name'})\n\n if [[ -n $result ]]; then\n echo \"$namespace namespace already exists in the cluster\"\n else\n echo \"$namespace namespace does not exist in the cluster\"\n echo \"Creating $namespace namespace in the cluster...\"\n kubectl create namespace $namespace\n fi\n\n # Create service account\n echo \"Creating $serviceAccountName service account...\"\n cat <<EOF | kubectl apply -f -\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n annotations:\n azure.workload.identity/client-id: $workloadManagedIdentityClientId\n azure.workload.identity/tenant-id: $tenantId\n labels:\n azure.workload.identity/use: \"true\"\n name: $serviceAccountName\n namespace: $namespace\nEOF\n fi\n\n if [[ \"$applicationGatewayForContainersEnabled\" == \"true\" \\\n && -n \"$applicationGatewayForContainersManagedIdentityClientId\" \\\n && -n \"$applicationGatewayForContainersSubnetId\" ]]; then\n \n # Install the Application Load Balancer\n echo \"Installing Application Load Balancer Controller in $applicationGatewayForContainersNamespace namespace using $applicationGatewayForContainersManagedIdentityClientId managed identity...\"\n helm upgrade alb-controller oci://mcr.microsoft.com/application-lb/charts/alb-controller \\\n --install \\\n --create-namespace \\\n --namespace $applicationGatewayForContainersNamespace \\\n --version 1.0.0 \\\n --set albController.namespace=$applicationGatewayForContainersNamespace \\\n --set albController.podIdentity.clientID=$applicationGatewayForContainersManagedIdentityClientId\n \n if [[ $? == 0 ]]; then\n echo \"Application Load Balancer Controller successfully installed\"\n else\n echo \"Failed to install Application Load Balancer Controller\"\n exit -1\n fi\n\n if [[ \"$applicationGatewayForContainersType\" == \"managed\" ]]; then\n # Create alb-infra namespace\n albInfraNamespace='alb-infra'\n result=$(kubectl get namespace -o 'jsonpath={.items[?(@.metadata.name==\"'$albInfraNamespace'\")].metadata.name'})\n\n if [[ -n $result ]]; then\n echo \"$albInfraNamespace namespace already exists in the cluster\"\n else\n echo \"$albInfraNamespace namespace does not exist in the cluster\"\n echo \"Creating $albInfraNamespace namespace in the cluster...\"\n kubectl create namespace $albInfraNamespace\n fi\n\n # Define the ApplicationLoadBalancer resource, specifying the subnet ID the Application Gateway for Containers association resource should deploy into. \n # The association establishes connectivity from Application Gateway for Containers to the defined subnet (and connected networks where applicable) to \n # be able to proxy traffic to a defined backend.\n echo \"Creating ApplicationLoadBalancer resource...\"\n kubectl apply -f - <<EOF\napiVersion: alb.networking.azure.io/v1\nkind: ApplicationLoadBalancer\nmetadata:\n name: alb\n namespace: alb-infra\nspec:\n associations:\n - $applicationGatewayForContainersSubnetId\nEOF\n if [[ -n $nodeResourceGroupName ]]; then \\\n echo -n \"Retrieving the resource id of the Application Gateway for Containers...\"\n counter=1\n while [ $counter -le 20 ]\n do\n # Retrieve the resource id of the managed Application Gateway for Containers resource\n applicationGatewayForContainersId=$(az resource list \\\n --resource-type \"Microsoft.ServiceNetworking/TrafficControllers\" \\\n --resource-group $nodeResourceGroupName \\\n --query [0].id \\\n --output tsv)\n if [[ -n $applicationGatewayForContainersId ]]; then\n echo \n break \n else\n echo -n '.'\n counter=$((counter + 1))\n sleep 1\n fi\n done\n\n if [[ -n $applicationGatewayForContainersId ]]; then\n applicationGatewayForContainersName=$(basename $applicationGatewayForContainersId)\n echo \"[$applicationGatewayForContainersId] resource id of the [$applicationGatewayForContainersName] Application Gateway for Containers successfully retrieved\"\n else\n echo \"Failed to retrieve the resource id of the Application Gateway for Containers\"\n exit -1\n fi\n\n # Check if the diagnostic setting already exists for the Application Gateway for Containers\n echo \"Checking if the [$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers actually exists...\"\n result=$(az monitor diagnostic-settings show \\\n --name $diagnosticSettingName \\\n --resource $applicationGatewayForContainersId \\\n --query name \\\n --output tsv 2>/dev/null)\n\n if [[ -z $result ]]; then\n echo \"[$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers does not exist\"\n echo \"Creating [$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers...\"\n\n # Create the diagnostic setting for the Application Gateway for Containers\n az monitor diagnostic-settings create \\\n --name $diagnosticSettingName \\\n --resource $applicationGatewayForContainersId \\\n --logs '[{\"categoryGroup\": \"allLogs\", \"enabled\": true}]' \\\n --metrics '[{\"category\": \"AllMetrics\", \"enabled\": true}]' \\\n --workspace $workspaceId \\\n --only-show-errors 1>/dev/null\n\n if [[ $? == 0 ]]; then\n echo \"[$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers successfully created\"\n else\n echo \"Failed to create [$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers\"\n exit -1\n fi\n else\n echo \"[$diagnosticSettingName] diagnostic setting for the [$applicationGatewayForContainersName] Application Gateway for Containers already exists\"\n fi\n fi\n fi\n fi\nfi\n\n# Create output as JSON file\necho '{}' |\n jq --arg x $applicationGatewayForContainersName '.applicationGatewayForContainersName=$x' |\n jq --arg x $namespace '.namespace=$x' |\n jq --arg x $serviceAccountName '.serviceAccountName=$x' |\n jq --arg x 'prometheus' '.prometheus=$x' |\n jq --arg x 'cert-manager' '.certManager=$x' |\n jq --arg x 'ingress-basic' '.nginxIngressController=$x' >$AZ_SCRIPTS_OUTPUT_PATH
\n
\n
The script performs the following steps:
\n
\n\n
Installskubectlusing the Azure CLI commandaz aks install-cli.
\n
Retrieves the AKS cluster credentials using the Azure CLI commandaz aks get-credentials.
\n
Checks whether the AKS cluster is private or public by querying theenablePrivateClusterattribute of the cluster's API server access profile using the Azure CLI commandaz aks show.
\n
InstallsHelmby downloading and executing theget_helm.shscript.
Updates the Helm repositories using thehelm repo updatecommand.
\n
Initializes variables related to the Application Gateway for Containers.
\n
The script performs the subsequent steps differently depending on whether the cluster is public or private. The script uses theaz aks command invoketo execute commands when the cluster is private.
\n
Installs Prometheus using thehelm upgrade --installcommand.
\n
Installs the certificate manager using thehelm upgrade --installcommand.
\n
If anamespaceandserviceAccountNameare provided, it creates the namespace and service account usingkubectl. This information is optional, and it can be used to create the namespace and service account for a workload.
\n
If the Application Gateway for Containers is enabled and necessary information is provided, it installs the Application Load Balancer Controller using thehelm upgrade --installcommand. The YAML manifest specifies the client id of the ALB Controller managed identity from theapplicationGatewayForContainersManagedIdentityClientIdenvironment variable and the target namespace from theapplicationGatewayForContainersNamespaceenvironment variable. For more information on the installation of the ALB Controller via Helm, seeQuickstart: Deploy Application Gateway for Containers ALB Controller.
\n
When theapplicationGatewayForContainersTypeenvironment variable is set tomanaged, creates thealb-infranamespace usingkubectland deploys theApplicationLoadBalancerresource in the newly created namespace. The YAML manifest specifies the resource id of the subnet used by the association from theapplicationGatewayForContainersSubnetIdenvironment variable.
\n
Retrieves the resource ID of the Application Gateway for Containers and checks if the diagnostic settings exist. If not, it creates the diagnostic settings usingaz monitor diagnostic-settings create.
\n
Creates an output JSON file containing the Application Gateway for Containersname, worloadnamespaceandservice account name, if any, and the namespace for Prometheus and the Certificate Manager.
\n\n
\n
Review deployed resources
\n
You can use the Azure portal to list the deployed resources in the resource group. If you chose to deploy an Application Gateway for Containers managed by the ALB Controller, you will find the resource under the node resource group of the AKS cluster.
\n
\n
\n
\n
You can also use Azure CLI to list the deployed resources in the resource group and AKS node resource group:
\n
\n
az resource list --resource-group <resource-group-name>\nnodeResourceGroupName=$(az aks show --name <aks-name> --resource-group <resource-group-name> --query \"nodeResourceGroup\" -o tsv)\naz resource list --resource-group nodeResourceGroupName
\n
\n
You can also use the following PowerShell cmdlet to list the deployed resources in the resource group and AKS node resource group:
After confirming the successful deployment, you can easily deploy your workloads and configure them to have a public endpoint through the newly created Application Gateway for Containers. To achieve this, you have two options: either aGatewayor anIngressin Kubernetes. These options allow you to expose your application to the public internet using the Application Gateway for Containers resource. The documentation provides various tutorials for both the Gateway API and the Ingress API, including:
You can find the scripts and YAML manifests for these tutorials in thetutorialsfolder. Additionally, theappfolder contains two samples:byofor a bring-your-own installation of the Application Gateway for Containers, andmanagedwhich works with Application Gateway for Containers managed by the ALB Controller.
\n
For simplicity, let's focus on themanagedsample, while leaving thebyosample for the reader to review. Let's start by reviewing the YAML manifests.
\n
\n
Deployment
\n
Thedeployment.yamlfile contains the YAML definition for the deployment, the service, and a secret that contains a temporary certificate for the Gatewy listener that will be replaced by the certificate issued byLet's Encryptvia the certificate manager. For more information on how to use the certificate manager to issue a new certificate to a Gateway using HTTP01 challenges, seeConfiguring the HTTP-01 Gateway API solver.
Thegateway.yamlcontains the definition of the Gateway used by the application. When using an Application Gateway for Containers managed by the ALB Controller, the frontend is auotmatically created for your by the ALB Controller.
The Gateway defines a certificate issuer in thecert-manager.io/issuerannotation, so we need to create an issuer. In the issuer, we define the CA root,Let's Encryptin this case, for the certificate chain to issue our certificate and the challenge type that our client would like to handle to prove our controll over the domain (in our case we will use HTTP01 challenge).
The certificate manager follows a series of steps to issue a certificate to the Gateway listener:
\n
\n\n
The gateway creates a Certificate object with a nil revision flag, initially pointing to a self-signed TLS secret. However, since the CA of the TLS secret is not valid, the Certificate realizes this and proceeds to the next step.
\n
A CertificateRequest (CR) object is created with the revision flag set to 1, indicating the need to re-issue a valid certificate. The CR contains all the necessary information to send a CSR request in PKCS #10 format to the CA.
\n
The CR object creates an Order object to monitor the request process.
\n
The Cluster Issuer registers itself in the ACME server (CA) using our public key, which is included in the CR. The CA server generates a unique token/key for our request and associates them with our public key, enabling the verification of our signature for future requests from our client.
\n
The CA server returns the unique token and key for each supported challenge to our client (Cluster Issuer), which were stored for our public key on the CA server.
\n
The cluster issuer updates the Order object with the server-supported challenges, along with their unique token and key.
\n
Based on the supported challenges and our Issuer configuration, the Order object determines to solve the HTTP01 Challenge, utilizing the parameters provided by the ACME server.
\n
A new pod is created in the default namespace to run the Challenge. This pod contains an HTTP server that serves on a specific path and expects the origin to be our domain name. If the origin does not match, a 404 error is returned.
\n\n
\n
HTTPRoute
\n
Thehttproute.yamlcontains the definition of theHTTPRouteobject used to route requests to the service:
Run the01-install-cert-manager.shscript if you need to install the certificate manager in your AKS cluster.
\n
\n
#/bin/bash\n\n# Variables\nsource ./00-variables.sh\n\n# Check if the cert-manager repository is not already added\nresult=$(helm repo list | grep $cmRepoName | awk '{print $1}')\n\nif [[ -n $result ]]; then\n echo \"[$cmRepoName] Helm repo already exists\"\nelse\n # Add the Jetstack Helm repository\n echo \"Adding [$cmRepoName] Helm repo...\"\n helm repo add $cmRepoName $cmRepoUrl\nfi\n\n# Update your local Helm chart repository cache\necho 'Updating Helm repos...'\nhelm repo update\n\n# Install cert-manager Helm chart\nresult=$(helm list -n $cmNamespace | grep $cmReleaseName | awk '{print $1}')\n\nif [[ -n $result ]]; then\n echo \"[$cmReleaseName] cert-manager already exists in the $cmNamespace namespace\"\n echo \"Upgrading [$cmReleaseName] cert-manager to the $cmNamespace namespace...\"\nelse\n # Install the cert-manager Helm chart\n echo \"Deploying [$cmReleaseName] cert-manager to the $cmNamespace namespace...\"\nfi\n\nhelm upgrade $cmReleaseName $cmRepoName/$cmChartName \\\n --install \\\n --create-namespace \\\n --namespace $cmNamespace \\\n --version $cmVersion \\\n --set installCRDs=true \\\n --set nodeSelector.\"kubernetes\\.io/os\"=linux \\\n --set \"extraArgs={--feature-gates=ExperimentalGatewayAPISupport=true}\"
\n
\n
Then run the02-create-sample.shscript to deploy the application to the specified namespace. The script makes use of theyqtool.
\n
\n
#!/bin/bash\n\n# Variables\nsource ./00-variables.sh\n\n# Check if namespace exists in the cluster\nresult=$(kubectl get namespace -o jsonpath=\"{.items[?(@.metadata.name=='$namespace')].metadata.name}\")\n\nif [[ -n $result ]]; then\n echo \"$namespace namespace already exists in the cluster\"\nelse\n echo \"$namespace namespace does not exist in the cluster\"\n echo \"creating $namespace namespace in the cluster...\"\n kubectl create namespace $namespace\nfi\n\n# Create a sample web application\nkubectl apply -n $namespace -f ./deployment.yaml\n\n# Create Gateway\ncat gateway.yaml |\n yq \"(.metadata.name)|=\"\\\"\"$gatewayName\"\\\" |\n yq \"(.metadata.namespace)|=\"\\\"\"$namespace\"\\\" |\n yq \"(.metadata.annotations.\"\\\"\"cert-manager.io/issuer\"\\\"\")|=\"\\\"\"$issuerName\"\\\" |\n yq \"(.metadata.annotations.\"\\\"\"alb.networking.azure.io/alb-name\"\\\"\")|=\"\\\"\"$applicationLoadBalancerName\"\\\" |\n yq \"(.metadata.annotations.\"\\\"\"alb.networking.azure.io/alb-namespace\"\\\"\")|=\"\\\"\"$applicationLoadBalancerNamespace\"\\\" |\n yq \"(.spec.listeners[1].hostname)|=\"\\\"\"$hostname\"\\\" |\nkubectl apply -f -\n\n# Create Issuer\ncat issuer.yaml |\n yq \"(.metadata.name)|=\"\\\"\"$issuerName\"\\\" |\n yq \"(.metadata.namespace)|=\"\\\"\"$namespace\"\\\" |\n yq \"(.spec.acme.solvers[0].http01.gatewayHTTPRoute.parentRefs[0].name)|=\"\\\"\"$gatewayName\"\\\" |\n yq \"(.spec.acme.solvers[0].http01.gatewayHTTPRoute.parentRefs[0].namespace)|=\"\\\"\"$namespace\"\\\" |\nkubectl apply -f -\n\n# Create HTTPRoute\ncat httproute.yaml |\n yq \"(.metadata.name)|=\"\\\"\"$httpRouteName\"\\\" |\n yq \"(.metadata.namespace)|=\"\\\"\"$namespace\"\\\" |\n yq \"(.spec.parentRefs[0].name)|=\"\\\"\"$gatewayName\"\\\" |\n yq \"(.spec.parentRefs[0].namespace)|=\"\\\"\"$namespace\"\\\" |\nkubectl apply -f -
\n
\n
If you delegated the management of your public DNS to Azure DNS, you can use the03-configure-dns.shscript to create a CNAME for the FQDN assigned to the frontend used by the Gateway.
\n
\n
# Variables\nsource ./00-variables.sh\n\n# Get the FQDN of the gateway\necho -n \"Retrieving the FQDN of the [$gatewayName] gateway...\"\nwhile true\ndo\n fqdn=$(kubectl get gateway $gatewayName -n $namespace -o jsonpath='{.status.addresses[0].value}')\n if [[ -n $fqdn ]]; then\n echo \n break \n else\n echo -n '.'\n sleep 1\n fi\ndone\n\nif [ -n $fqdn ]; then\n echo \"[$fqdn] FQDN successfully retrieved from the [$gatewayName] gateway\"\nelse\n echo \"Failed to retrieve the FQDN from the [$gatewayName] gateway\"\n exit\nfi\n\n# Check if an CNAME record for todolist subdomain exists in the DNS Zone\necho \"Retrieving the CNAME for the [$subdomain] subdomain from the [$dnsZoneName] DNS zone...\"\ncname=$(az network dns record-set cname list \\\n --zone-name $dnsZoneName \\\n --resource-group $dnsZoneResourceGroupName \\\n --query \"[?name=='$subdomain'].CNAMERecord.cname\" \\\n --output tsv \\\n --only-show-errors)\n\nif [[ -n $cname ]]; then\n echo \"A CNAME already exists in [$dnsZoneName] DNS zone for the [$subdomain]\"\n\n if [[ $cname == $fqdn ]]; then\n echo \"The [$cname] CNAME equals the FQDN of the [$gatewayName] gateway. No additional step is required.\"\n exit\n else\n echo \"The [$cname] CNAME is different than the [$fqdn] FQDN of the [$gatewayName] gateway\"\n fi\n\n # Delete the CNAME record\n echo \"Deleting the [$subdomain] CNAME from the [$dnsZoneName] zone...\"\n\n az network dns record-set cname delete \\\n --name $subdomain \\\n --zone-name $dnsZoneName \\\n --resource-group $dnsZoneResourceGroupName \\\n --only-show-errors \\\n --yes\n\n if [[ $? == 0 ]]; then\n echo \"[$subdomain] CNAME successfully deleted from the [$dnsZoneName] zone\"\n else\n echo \"Failed to delete the [$subdomain] CNAME from the [$dnsZoneName] zone\"\n exit\n fi\nelse\n echo \"No CNAME exists in [$dnsZoneName] DNS zone for the [$subdomain] subdomain\"\nfi\n\n# Create a CNAME record\necho \"Creating a CNAME in the [$dnsZoneName] DNS zone for the [$fqdn] FQDN of the [$gatewayName] gateway...\"\naz network dns record-set cname set-record \\\n --cname $fqdn \\\n --zone-name $dnsZoneName \\\n --resource-group $dnsZoneResourceGroupName \\\n --record-set-name $subdomain \\\n --only-show-errors 1>/dev/null\n\nif [[ $? == 0 ]]; then\n echo \"[$subdomain] CNAME successfully created in the [$dnsZoneName] DNS zone for the [$fqdn] FQDN of the [$gatewayName] gateway\"\nelse\n echo \"Failed to create a CNAME in the [$dnsZoneName] DNS zone for the [$fqdn] FQDN of the [$gatewayName] gateway\"\nfi
\n
\n
Finally, you can test the sample by running the04-test-application.shscript.
\n
\n
\n
\n
\n
\n#!/bin/bash\n\n# Variables\nsource ./00-variables.sh\n\n# Curling this FQDN should return responses from the backend as configured in the HTTPRoute\ncurl https://$hostname\n\n
\n
\n
\n
\n
\n
\n
You can also open the application using a web browser:
\n
\n
\n
\n
Clean up resources
\n
You can delete the resource group using the following Azure CLI command when you no longer need the resources you created. This will remove all the Azure resources.
\n
\n
az group delete --name <resource-group-name>
\n
\n
Alternatively, you can use the following PowerShell cmdlet to delete the resource group and all the Azure resources.
This article provides a detailed guide on installing an AKS (Azure Kubernetes Service) cluster and Azure Application Gateway for Containers, all using Bicep.
\n
\n
\n
","introduction":"","coverImage":null,"coverImageProperties":{"__typename":"CoverImageProperties","style":"STANDARD","titlePosition":"BOTTOM","altText":""},"currentRevision":{"__ref":"Revision:revision:3967434_12"},"latestVersion":{"__typename":"FriendlyVersion","major":"12","minor":"0"},"metrics":{"__typename":"MessageMetrics","views":17917},"visibilityScope":"PUBLIC","canonicalUrl":null,"seoTitle":null,"seoDescription":null,"placeholder":false,"originalMessageForPlaceholder":null,"contributors":{"__typename":"UserConnection","edges":[]},"nonCoAuthorContributors":{"__typename":"UserConnection","edges":[]},"coAuthors":{"__typename":"UserConnection","edges":[]},"blogMessagePolicies":{"__typename":"BlogMessagePolicies","canDoAuthoringActionsOnBlog":{"__typename":"PolicyResult","failureReason":{"__typename":"FailureReason","message":"error.lithium.policies.blog.action_can_do_authoring_action.accessDenied","key":"error.lithium.policies.blog.action_can_do_authoring_action.accessDenied","args":[]}}},"archivalData":null,"replies":{"__typename":"MessageConnection","edges":[{"__typename":"MessageEdge","cursor":"MjUuMXwyLjF8aXwxMHwxMzI6MHxpbnQsMzk4NDUxNSwzOTg0NTE1","node":{"__ref":"BlogReplyMessage:message:3984515"}},{"__typename":"MessageEdge","cursor":"MjUuMXwyLjF8aXwxMHwxMzI6MHxpbnQsMzk4NDUxNSwzOTY4NTQ0","node":{"__ref":"BlogReplyMessage:message:3968544"}},{"__typename":"MessageEdge","cursor":"MjUuMXwyLjF8aXwxMHwxMzI6MHxpbnQsMzk4NDUxNSwzOTY3NDc5","node":{"__ref":"BlogReplyMessage:message:3967479"}},{"__typename":"MessageEdge","cursor":"MjUuMXwyLjF8aXwxMHwxMzI6MHxpbnQsMzk4NDUxNSwzOTY3NDU3","node":{"__ref":"BlogReplyMessage:message:3967457"}}],"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}},"customFields":[],"revisions({\"constraints\":{\"isPublished\":{\"eq\":true}},\"first\":1})":{"__typename":"RevisionConnection","totalCount":12}},"Conversation:conversation:3967434":{"__typename":"Conversation","id":"conversation:3967434","solved":false,"topic":{"__ref":"BlogTopicMessage:message:3967434"},"lastPostingActivityTime":"2024-05-02T00:21:23.582-07:00","lastPostTime":"2023-11-15T23:30:55.019-08:00","unreadReplyCount":4,"isSubscribed":false},"ModerationData:moderation_data:3967434":{"__typename":"ModerationData","id":"moderation_data:3967434","status":"APPROVED","rejectReason":null,"isReportedAbuse":false,"rejectUser":null,"rejectTime":null,"rejectActorType":null},"AssociatedImage:{\"url\":\"https://techcommunity.microsoft.com/t5/s/gxcuf89792/images/bS0zOTY3NDM0LTUyMDQ1MGkxRjQ3RjlFN0I5RDgwNjg4?revision=12\"}":{"__typename":"AssociatedImage","url":"https://techcommunity.microsoft.com/t5/s/gxcuf89792/images/bS0zOTY3NDM0LTUyMDQ1MGkxRjQ3RjlFN0I5RDgwNjg4?revision=12","title":"architecture.png","associationType":"TEASER","width":820,"height":1000,"altText":null},"AssociatedImage:{\"url\":\"https://techcommunity.microsoft.com/t5/s/gxcuf89792/images/bS0zOTY3NDM0LTUyMDQ4N2k3NjdEQUY5QkIyMTA3Rjcz?revision=12\"}":{"__typename":"AssociatedImage","url":"https://techcommunity.microsoft.com/t5/s/gxcuf89792/images/bS0zOTY3NDM0LTUyMDQ4N2k3NjdEQUY5QkIyMTA3Rjcz?revision=12","title":"architecture.png","associationType":"BODY","width":1400,"height":1000,"altText":null},"AssociatedImage:{\"url\":\"https://techcommunity.microsoft.com/t5/s/gxcuf89792/images/bS0zOTY3NDM0LTUyMDQ1M2lFQjEzNzRDQkVCN0U3RTdC?revision=12\"}":{"__typename":"AssociatedImage","url":"https://techcommunity.microsoft.com/t5/s/gxcuf89792/images/bS0zOTY3NDM0LTUyMDQ1M2lFQjEzNzRDQkVCN0U3RTdC?revision=12","title":"gateway-api.png","associationType":"BODY","width":1794,"height":1406,"altText":null},"AssociatedImage:{\"url\":\"https://techcommunity.microsoft.com/t5/s/gxcuf89792/images/bS0zOTY3NDM0LTUyMDQ2MWk3MUM2MEU2MEQzMDg1QkEx?revision=12\"}":{"__typename":"AssociatedImage","url":"https://techcommunity.microsoft.com/t5/s/gxcuf89792/images/bS0zOTY3NDM0LTUyMDQ2MWk3MUM2MEU2MEQzMDg1QkEx?revision=12","title":"node-resource-group.png","associationType":"BODY","width":798,"height":985,"altText":null},"AssociatedImage:{\"url\":\"https://techcommunity.microsoft.com/t5/s/gxcuf89792/images/bS0zOTY3NDM0LTUyMDQ2MmkxQTJDNzU2MkU3QzA5QTcy?revision=12\"}":{"__typename":"AssociatedImage","url":"https://techcommunity.microsoft.com/t5/s/gxcuf89792/images/bS0zOTY3NDM0LTUyMDQ2MmkxQTJDNzU2MkU3QzA5QTcy?revision=12","title":"sample.png","associationType":"BODY","width":1256,"height":1146,"altText":null},"Revision:revision:3967434_12":{"__typename":"Revision","id":"revision:3967434_12","lastEditTime":"2024-05-02T00:21:23.582-07:00"},"CachedAsset:theme:customTheme1-1742487241985":{"__typename":"CachedAsset","id":"theme:customTheme1-1742487241985","value":{"id":"customTheme1","animation":{"fast":"150ms","normal":"250ms","slow":"500ms","slowest":"750ms","function":"cubic-bezier(0.07, 0.91, 0.51, 1)","__typename":"AnimationThemeSettings"},"avatar":{"borderRadius":"50%","collections":["default"],"__typename":"AvatarThemeSettings"},"basics":{"browserIcon":{"imageAssetName":"favicon-1730836283320.png","imageLastModified":"1730836286415","__typename":"ThemeAsset"},"customerLogo":{"imageAssetName":"favicon-1730836271365.png","imageLastModified":"1730836274203","__typename":"ThemeAsset"},"maximumWidthOfPageContent":"1300px","oneColumnNarrowWidth":"800px","gridGutterWidthMd":"30px","gridGutterWidthXs":"10px","pageWidthStyle":"WIDTH_OF_BROWSER","__typename":"BasicsThemeSettings"},"buttons":{"borderRadiusSm":"3px","borderRadius":"3px","borderRadiusLg":"5px","paddingY":"5px","paddingYLg":"7px","paddingYHero":"var(--lia-bs-btn-padding-y-lg)","paddingX":"12px","paddingXLg":"16px","paddingXHero":"60px","fontStyle":"NORMAL","fontWeight":"700","textTransform":"NONE","disabledOpacity":0.5,"primaryTextColor":"var(--lia-bs-white)","primaryTextHoverColor":"var(--lia-bs-white)","primaryTextActiveColor":"var(--lia-bs-white)","primaryBgColor":"var(--lia-bs-primary)","primaryBgHoverColor":"hsl(var(--lia-bs-primary-h), var(--lia-bs-primary-s), calc(var(--lia-bs-primary-l) * 0.85))","primaryBgActiveColor":"hsl(var(--lia-bs-primary-h), var(--lia-bs-primary-s), calc(var(--lia-bs-primary-l) * 0.7))","primaryBorder":"1px solid transparent","primaryBorderHover":"1px solid transparent","primaryBorderActive":"1px solid transparent","primaryBorderFocus":"1px solid var(--lia-bs-white)","primaryBoxShadowFocus":"0 0 0 1px var(--lia-bs-primary), 0 0 0 4px hsla(var(--lia-bs-primary-h), var(--lia-bs-primary-s), var(--lia-bs-primary-l), 0.2)","secondaryTextColor":"var(--lia-bs-gray-900)","secondaryTextHoverColor":"hsl(var(--lia-bs-gray-900-h), var(--lia-bs-gray-900-s), calc(var(--lia-bs-gray-900-l) * 0.95))","secondaryTextActiveColor":"hsl(var(--lia-bs-gray-900-h), var(--lia-bs-gray-900-s), calc(var(--lia-bs-gray-900-l) * 0.9))","secondaryBgColor":"var(--lia-bs-gray-200)","secondaryBgHoverColor":"hsl(var(--lia-bs-gray-200-h), var(--lia-bs-gray-200-s), calc(var(--lia-bs-gray-200-l) * 0.96))","secondaryBgActiveColor":"hsl(var(--lia-bs-gray-200-h), var(--lia-bs-gray-200-s), calc(var(--lia-bs-gray-200-l) * 0.92))","secondaryBorder":"1px solid transparent","secondaryBorderHover":"1px solid transparent","secondaryBorderActive":"1px solid transparent","secondaryBorderFocus":"1px solid transparent","secondaryBoxShadowFocus":"0 0 0 1px var(--lia-bs-primary), 0 0 0 4px hsla(var(--lia-bs-primary-h), var(--lia-bs-primary-s), var(--lia-bs-primary-l), 0.2)","tertiaryTextColor":"var(--lia-bs-gray-900)","tertiaryTextHoverColor":"hsl(var(--lia-bs-gray-900-h), var(--lia-bs-gray-900-s), calc(var(--lia-bs-gray-900-l) * 0.95))","tertiaryTextActiveColor":"hsl(var(--lia-bs-gray-900-h), var(--lia-bs-gray-900-s), calc(var(--lia-bs-gray-900-l) * 0.9))","tertiaryBgColor":"transparent","tertiaryBgHoverColor":"transparent","tertiaryBgActiveColor":"hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.04)","tertiaryBorder":"1px solid transparent","tertiaryBorderHover":"1px solid hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.08)","tertiaryBorderActive":"1px solid transparent","tertiaryBorderFocus":"1px solid transparent","tertiaryBoxShadowFocus":"0 0 0 1px var(--lia-bs-primary), 0 0 0 4px hsla(var(--lia-bs-primary-h), var(--lia-bs-primary-s), var(--lia-bs-primary-l), 0.2)","destructiveTextColor":"var(--lia-bs-danger)","destructiveTextHoverColor":"hsl(var(--lia-bs-danger-h), var(--lia-bs-danger-s), calc(var(--lia-bs-danger-l) * 0.95))","destructiveTextActiveColor":"hsl(var(--lia-bs-danger-h), var(--lia-bs-danger-s), calc(var(--lia-bs-danger-l) * 0.9))","destructiveBgColor":"var(--lia-bs-gray-200)","destructiveBgHoverColor":"hsl(var(--lia-bs-gray-200-h), var(--lia-bs-gray-200-s), calc(var(--lia-bs-gray-200-l) * 0.96))","destructiveBgActiveColor":"hsl(var(--lia-bs-gray-200-h), var(--lia-bs-gray-200-s), calc(var(--lia-bs-gray-200-l) * 0.92))","destructiveBorder":"1px solid transparent","destructiveBorderHover":"1px solid transparent","destructiveBorderActive":"1px solid transparent","destructiveBorderFocus":"1px solid transparent","destructiveBoxShadowFocus":"0 0 0 1px var(--lia-bs-primary), 0 0 0 4px hsla(var(--lia-bs-primary-h), var(--lia-bs-primary-s), var(--lia-bs-primary-l), 0.2)","__typename":"ButtonsThemeSettings"},"border":{"color":"hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.08)","mainContent":"NONE","sideContent":"LIGHT","radiusSm":"3px","radius":"5px","radiusLg":"9px","radius50":"100vw","__typename":"BorderThemeSettings"},"boxShadow":{"xs":"0 0 0 1px hsla(var(--lia-bs-gray-900-h), var(--lia-bs-gray-900-s), var(--lia-bs-gray-900-l), 0.08), 0 3px 0 -1px hsla(var(--lia-bs-gray-900-h), var(--lia-bs-gray-900-s), var(--lia-bs-gray-900-l), 0.16)","sm":"0 2px 4px hsla(var(--lia-bs-gray-900-h), var(--lia-bs-gray-900-s), var(--lia-bs-gray-900-l), 0.12)","md":"0 5px 15px hsla(var(--lia-bs-gray-900-h), var(--lia-bs-gray-900-s), var(--lia-bs-gray-900-l), 0.3)","lg":"0 10px 30px hsla(var(--lia-bs-gray-900-h), var(--lia-bs-gray-900-s), var(--lia-bs-gray-900-l), 0.3)","__typename":"BoxShadowThemeSettings"},"cards":{"bgColor":"var(--lia-panel-bg-color)","borderRadius":"var(--lia-panel-border-radius)","boxShadow":"var(--lia-box-shadow-xs)","__typename":"CardsThemeSettings"},"chip":{"maxWidth":"300px","height":"30px","__typename":"ChipThemeSettings"},"coreTypes":{"defaultMessageLinkColor":"var(--lia-bs-link-color)","defaultMessageLinkDecoration":"none","defaultMessageLinkFontStyle":"NORMAL","defaultMessageLinkFontWeight":"400","defaultMessageFontStyle":"NORMAL","defaultMessageFontWeight":"400","forumColor":"#4099E2","forumFontFamily":"var(--lia-bs-font-family-base)","forumFontWeight":"var(--lia-default-message-font-weight)","forumLineHeight":"var(--lia-bs-line-height-base)","forumFontStyle":"var(--lia-default-message-font-style)","forumMessageLinkColor":"var(--lia-default-message-link-color)","forumMessageLinkDecoration":"var(--lia-default-message-link-decoration)","forumMessageLinkFontStyle":"var(--lia-default-message-link-font-style)","forumMessageLinkFontWeight":"var(--lia-default-message-link-font-weight)","forumSolvedColor":"#148563","blogColor":"#1CBAA0","blogFontFamily":"var(--lia-bs-font-family-base)","blogFontWeight":"var(--lia-default-message-font-weight)","blogLineHeight":"1.75","blogFontStyle":"var(--lia-default-message-font-style)","blogMessageLinkColor":"var(--lia-default-message-link-color)","blogMessageLinkDecoration":"var(--lia-default-message-link-decoration)","blogMessageLinkFontStyle":"var(--lia-default-message-link-font-style)","blogMessageLinkFontWeight":"var(--lia-default-message-link-font-weight)","tkbColor":"#4C6B90","tkbFontFamily":"var(--lia-bs-font-family-base)","tkbFontWeight":"var(--lia-default-message-font-weight)","tkbLineHeight":"1.75","tkbFontStyle":"var(--lia-default-message-font-style)","tkbMessageLinkColor":"var(--lia-default-message-link-color)","tkbMessageLinkDecoration":"var(--lia-default-message-link-decoration)","tkbMessageLinkFontStyle":"var(--lia-default-message-link-font-style)","tkbMessageLinkFontWeight":"var(--lia-default-message-link-font-weight)","qandaColor":"#4099E2","qandaFontFamily":"var(--lia-bs-font-family-base)","qandaFontWeight":"var(--lia-default-message-font-weight)","qandaLineHeight":"var(--lia-bs-line-height-base)","qandaFontStyle":"var(--lia-default-message-link-font-style)","qandaMessageLinkColor":"var(--lia-default-message-link-color)","qandaMessageLinkDecoration":"var(--lia-default-message-link-decoration)","qandaMessageLinkFontStyle":"var(--lia-default-message-link-font-style)","qandaMessageLinkFontWeight":"var(--lia-default-message-link-font-weight)","qandaSolvedColor":"#3FA023","ideaColor":"#FF8000","ideaFontFamily":"var(--lia-bs-font-family-base)","ideaFontWeight":"var(--lia-default-message-font-weight)","ideaLineHeight":"var(--lia-bs-line-height-base)","ideaFontStyle":"var(--lia-default-message-font-style)","ideaMessageLinkColor":"var(--lia-default-message-link-color)","ideaMessageLinkDecoration":"var(--lia-default-message-link-decoration)","ideaMessageLinkFontStyle":"var(--lia-default-message-link-font-style)","ideaMessageLinkFontWeight":"var(--lia-default-message-link-font-weight)","contestColor":"#FCC845","contestFontFamily":"var(--lia-bs-font-family-base)","contestFontWeight":"var(--lia-default-message-font-weight)","contestLineHeight":"var(--lia-bs-line-height-base)","contestFontStyle":"var(--lia-default-message-link-font-style)","contestMessageLinkColor":"var(--lia-default-message-link-color)","contestMessageLinkDecoration":"var(--lia-default-message-link-decoration)","contestMessageLinkFontStyle":"ITALIC","contestMessageLinkFontWeight":"var(--lia-default-message-link-font-weight)","occasionColor":"#D13A1F","occasionFontFamily":"var(--lia-bs-font-family-base)","occasionFontWeight":"var(--lia-default-message-font-weight)","occasionLineHeight":"var(--lia-bs-line-height-base)","occasionFontStyle":"var(--lia-default-message-font-style)","occasionMessageLinkColor":"var(--lia-default-message-link-color)","occasionMessageLinkDecoration":"var(--lia-default-message-link-decoration)","occasionMessageLinkFontStyle":"var(--lia-default-message-link-font-style)","occasionMessageLinkFontWeight":"var(--lia-default-message-link-font-weight)","grouphubColor":"#333333","categoryColor":"#949494","communityColor":"#FFFFFF","productColor":"#949494","__typename":"CoreTypesThemeSettings"},"colors":{"black":"#000000","white":"#FFFFFF","gray100":"#F7F7F7","gray200":"#F7F7F7","gray300":"#E8E8E8","gray400":"#D9D9D9","gray500":"#CCCCCC","gray600":"#717171","gray700":"#707070","gray800":"#545454","gray900":"#333333","dark":"#545454","light":"#F7F7F7","primary":"#0069D4","secondary":"#333333","bodyText":"#333333","bodyBg":"#FFFFFF","info":"#409AE2","success":"#41C5AE","warning":"#FCC844","danger":"#BC341B","alertSystem":"#FF6600","textMuted":"#707070","highlight":"#FFFCAD","outline":"var(--lia-bs-primary)","custom":["#D3F5A4","#243A5E"],"__typename":"ColorsThemeSettings"},"divider":{"size":"3px","marginLeft":"4px","marginRight":"4px","borderRadius":"50%","bgColor":"var(--lia-bs-gray-600)","bgColorActive":"var(--lia-bs-gray-600)","__typename":"DividerThemeSettings"},"dropdown":{"fontSize":"var(--lia-bs-font-size-sm)","borderColor":"var(--lia-bs-border-color)","borderRadius":"var(--lia-bs-border-radius-sm)","dividerBg":"var(--lia-bs-gray-300)","itemPaddingY":"5px","itemPaddingX":"20px","headerColor":"var(--lia-bs-gray-700)","__typename":"DropdownThemeSettings"},"email":{"link":{"color":"#0069D4","hoverColor":"#0061c2","decoration":"none","hoverDecoration":"underline","__typename":"EmailLinkSettings"},"border":{"color":"#e4e4e4","__typename":"EmailBorderSettings"},"buttons":{"borderRadiusLg":"5px","paddingXLg":"16px","paddingYLg":"7px","fontWeight":"700","primaryTextColor":"#ffffff","primaryTextHoverColor":"#ffffff","primaryBgColor":"#0069D4","primaryBgHoverColor":"#005cb8","primaryBorder":"1px solid transparent","primaryBorderHover":"1px solid transparent","__typename":"EmailButtonsSettings"},"panel":{"borderRadius":"5px","borderColor":"#e4e4e4","__typename":"EmailPanelSettings"},"__typename":"EmailThemeSettings"},"emoji":{"skinToneDefault":"#ffcd43","skinToneLight":"#fae3c5","skinToneMediumLight":"#e2cfa5","skinToneMedium":"#daa478","skinToneMediumDark":"#a78058","skinToneDark":"#5e4d43","__typename":"EmojiThemeSettings"},"heading":{"color":"var(--lia-bs-body-color)","fontFamily":"Segoe UI","fontStyle":"NORMAL","fontWeight":"400","h1FontSize":"34px","h2FontSize":"32px","h3FontSize":"28px","h4FontSize":"24px","h5FontSize":"20px","h6FontSize":"16px","lineHeight":"1.3","subHeaderFontSize":"11px","subHeaderFontWeight":"500","h1LetterSpacing":"normal","h2LetterSpacing":"normal","h3LetterSpacing":"normal","h4LetterSpacing":"normal","h5LetterSpacing":"normal","h6LetterSpacing":"normal","subHeaderLetterSpacing":"2px","h1FontWeight":"var(--lia-bs-headings-font-weight)","h2FontWeight":"var(--lia-bs-headings-font-weight)","h3FontWeight":"var(--lia-bs-headings-font-weight)","h4FontWeight":"var(--lia-bs-headings-font-weight)","h5FontWeight":"var(--lia-bs-headings-font-weight)","h6FontWeight":"var(--lia-bs-headings-font-weight)","__typename":"HeadingThemeSettings"},"icons":{"size10":"10px","size12":"12px","size14":"14px","size16":"16px","size20":"20px","size24":"24px","size30":"30px","size40":"40px","size50":"50px","size60":"60px","size80":"80px","size120":"120px","size160":"160px","__typename":"IconsThemeSettings"},"imagePreview":{"bgColor":"var(--lia-bs-gray-900)","titleColor":"var(--lia-bs-white)","controlColor":"var(--lia-bs-white)","controlBgColor":"var(--lia-bs-gray-800)","__typename":"ImagePreviewThemeSettings"},"input":{"borderColor":"var(--lia-bs-gray-600)","disabledColor":"var(--lia-bs-gray-600)","focusBorderColor":"var(--lia-bs-primary)","labelMarginBottom":"10px","btnFontSize":"var(--lia-bs-font-size-sm)","focusBoxShadow":"0 0 0 3px hsla(var(--lia-bs-primary-h), var(--lia-bs-primary-s), var(--lia-bs-primary-l), 0.2)","checkLabelMarginBottom":"2px","checkboxBorderRadius":"3px","borderRadiusSm":"var(--lia-bs-border-radius-sm)","borderRadius":"var(--lia-bs-border-radius)","borderRadiusLg":"var(--lia-bs-border-radius-lg)","formTextMarginTop":"4px","textAreaBorderRadius":"var(--lia-bs-border-radius)","activeFillColor":"var(--lia-bs-primary)","__typename":"InputThemeSettings"},"loading":{"dotDarkColor":"hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.2)","dotLightColor":"hsla(var(--lia-bs-white-h), var(--lia-bs-white-s), var(--lia-bs-white-l), 0.5)","barDarkColor":"hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.06)","barLightColor":"hsla(var(--lia-bs-white-h), var(--lia-bs-white-s), var(--lia-bs-white-l), 0.4)","__typename":"LoadingThemeSettings"},"link":{"color":"var(--lia-bs-primary)","hoverColor":"hsl(var(--lia-bs-primary-h), var(--lia-bs-primary-s), calc(var(--lia-bs-primary-l) - 10%))","decoration":"none","hoverDecoration":"underline","__typename":"LinkThemeSettings"},"listGroup":{"itemPaddingY":"15px","itemPaddingX":"15px","borderColor":"var(--lia-bs-gray-300)","__typename":"ListGroupThemeSettings"},"modal":{"contentTextColor":"var(--lia-bs-body-color)","contentBg":"var(--lia-bs-white)","backgroundBg":"var(--lia-bs-black)","smSize":"440px","mdSize":"760px","lgSize":"1080px","backdropOpacity":0.3,"contentBoxShadowXs":"var(--lia-bs-box-shadow-sm)","contentBoxShadow":"var(--lia-bs-box-shadow)","headerFontWeight":"700","__typename":"ModalThemeSettings"},"navbar":{"position":"FIXED","background":{"attachment":null,"clip":null,"color":"var(--lia-bs-white)","imageAssetName":"","imageLastModified":"0","origin":null,"position":"CENTER_CENTER","repeat":"NO_REPEAT","size":"COVER","__typename":"BackgroundProps"},"backgroundOpacity":0.8,"paddingTop":"15px","paddingBottom":"15px","borderBottom":"1px solid var(--lia-bs-border-color)","boxShadow":"var(--lia-bs-box-shadow-sm)","brandMarginRight":"30px","brandMarginRightSm":"10px","brandLogoHeight":"30px","linkGap":"10px","linkJustifyContent":"flex-start","linkPaddingY":"5px","linkPaddingX":"10px","linkDropdownPaddingY":"9px","linkDropdownPaddingX":"var(--lia-nav-link-px)","linkColor":"var(--lia-bs-body-color)","linkHoverColor":"var(--lia-bs-primary)","linkFontSize":"var(--lia-bs-font-size-sm)","linkFontStyle":"NORMAL","linkFontWeight":"400","linkTextTransform":"NONE","linkLetterSpacing":"normal","linkBorderRadius":"var(--lia-bs-border-radius-sm)","linkBgColor":"transparent","linkBgHoverColor":"transparent","linkBorder":"none","linkBorderHover":"none","linkBoxShadow":"none","linkBoxShadowHover":"none","linkTextBorderBottom":"none","linkTextBorderBottomHover":"none","dropdownPaddingTop":"10px","dropdownPaddingBottom":"15px","dropdownPaddingX":"10px","dropdownMenuOffset":"2px","dropdownDividerMarginTop":"10px","dropdownDividerMarginBottom":"10px","dropdownBorderColor":"hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.08)","controllerBgHoverColor":"hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.1)","controllerIconColor":"var(--lia-bs-body-color)","controllerIconHoverColor":"var(--lia-bs-body-color)","controllerTextColor":"var(--lia-nav-controller-icon-color)","controllerTextHoverColor":"var(--lia-nav-controller-icon-hover-color)","controllerHighlightColor":"hsla(30, 100%, 50%)","controllerHighlightTextColor":"var(--lia-yiq-light)","controllerBorderRadius":"var(--lia-border-radius-50)","hamburgerColor":"var(--lia-nav-controller-icon-color)","hamburgerHoverColor":"var(--lia-nav-controller-icon-color)","hamburgerBgColor":"transparent","hamburgerBgHoverColor":"transparent","hamburgerBorder":"none","hamburgerBorderHover":"none","collapseMenuMarginLeft":"20px","collapseMenuDividerBg":"var(--lia-nav-link-color)","collapseMenuDividerOpacity":0.16,"__typename":"NavbarThemeSettings"},"pager":{"textColor":"var(--lia-bs-link-color)","textFontWeight":"var(--lia-font-weight-md)","textFontSize":"var(--lia-bs-font-size-sm)","__typename":"PagerThemeSettings"},"panel":{"bgColor":"var(--lia-bs-white)","borderRadius":"var(--lia-bs-border-radius)","borderColor":"var(--lia-bs-border-color)","boxShadow":"none","__typename":"PanelThemeSettings"},"popover":{"arrowHeight":"8px","arrowWidth":"16px","maxWidth":"300px","minWidth":"100px","headerBg":"var(--lia-bs-white)","borderColor":"var(--lia-bs-border-color)","borderRadius":"var(--lia-bs-border-radius)","boxShadow":"0 0.5rem 1rem hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.15)","__typename":"PopoverThemeSettings"},"prism":{"color":"#000000","bgColor":"#f5f2f0","fontFamily":"var(--font-family-monospace)","fontSize":"var(--lia-bs-font-size-base)","fontWeightBold":"var(--lia-bs-font-weight-bold)","fontStyleItalic":"italic","tabSize":2,"highlightColor":"#b3d4fc","commentColor":"#62707e","punctuationColor":"#6f6f6f","namespaceOpacity":"0.7","propColor":"#990055","selectorColor":"#517a00","operatorColor":"#906736","operatorBgColor":"hsla(0, 0%, 100%, 0.5)","keywordColor":"#0076a9","functionColor":"#d3284b","variableColor":"#c14700","__typename":"PrismThemeSettings"},"rte":{"bgColor":"var(--lia-bs-white)","borderRadius":"var(--lia-panel-border-radius)","boxShadow":" var(--lia-panel-box-shadow)","customColor1":"#bfedd2","customColor2":"#fbeeb8","customColor3":"#f8cac6","customColor4":"#eccafa","customColor5":"#c2e0f4","customColor6":"#2dc26b","customColor7":"#f1c40f","customColor8":"#e03e2d","customColor9":"#b96ad9","customColor10":"#3598db","customColor11":"#169179","customColor12":"#e67e23","customColor13":"#ba372a","customColor14":"#843fa1","customColor15":"#236fa1","customColor16":"#ecf0f1","customColor17":"#ced4d9","customColor18":"#95a5a6","customColor19":"#7e8c8d","customColor20":"#34495e","customColor21":"#000000","customColor22":"#ffffff","defaultMessageHeaderMarginTop":"40px","defaultMessageHeaderMarginBottom":"20px","defaultMessageItemMarginTop":"0","defaultMessageItemMarginBottom":"10px","diffAddedColor":"hsla(170, 53%, 51%, 0.4)","diffChangedColor":"hsla(43, 97%, 63%, 0.4)","diffNoneColor":"hsla(0, 0%, 80%, 0.4)","diffRemovedColor":"hsla(9, 74%, 47%, 0.4)","specialMessageHeaderMarginTop":"40px","specialMessageHeaderMarginBottom":"20px","specialMessageItemMarginTop":"0","specialMessageItemMarginBottom":"10px","__typename":"RteThemeSettings"},"tags":{"bgColor":"var(--lia-bs-gray-200)","bgHoverColor":"var(--lia-bs-gray-400)","borderRadius":"var(--lia-bs-border-radius-sm)","color":"var(--lia-bs-body-color)","hoverColor":"var(--lia-bs-body-color)","fontWeight":"var(--lia-font-weight-md)","fontSize":"var(--lia-font-size-xxs)","textTransform":"UPPERCASE","letterSpacing":"0.5px","__typename":"TagsThemeSettings"},"toasts":{"borderRadius":"var(--lia-bs-border-radius)","paddingX":"12px","__typename":"ToastsThemeSettings"},"typography":{"fontFamilyBase":"Segoe UI","fontStyleBase":"NORMAL","fontWeightBase":"400","fontWeightLight":"300","fontWeightNormal":"400","fontWeightMd":"500","fontWeightBold":"700","letterSpacingSm":"normal","letterSpacingXs":"normal","lineHeightBase":"1.5","fontSizeBase":"16px","fontSizeXxs":"11px","fontSizeXs":"12px","fontSizeSm":"14px","fontSizeLg":"20px","fontSizeXl":"24px","smallFontSize":"14px","customFonts":[{"source":"SERVER","name":"Segoe UI","styles":[{"style":"NORMAL","weight":"400","__typename":"FontStyleData"},{"style":"NORMAL","weight":"300","__typename":"FontStyleData"},{"style":"NORMAL","weight":"600","__typename":"FontStyleData"},{"style":"NORMAL","weight":"700","__typename":"FontStyleData"},{"style":"ITALIC","weight":"400","__typename":"FontStyleData"}],"assetNames":["SegoeUI-normal-400.woff2","SegoeUI-normal-300.woff2","SegoeUI-normal-600.woff2","SegoeUI-normal-700.woff2","SegoeUI-italic-400.woff2"],"__typename":"CustomFont"},{"source":"SERVER","name":"MWF Fluent Icons","styles":[{"style":"NORMAL","weight":"400","__typename":"FontStyleData"}],"assetNames":["MWFFluentIcons-normal-400.woff2"],"__typename":"CustomFont"}],"__typename":"TypographyThemeSettings"},"unstyledListItem":{"marginBottomSm":"5px","marginBottomMd":"10px","marginBottomLg":"15px","marginBottomXl":"20px","marginBottomXxl":"25px","__typename":"UnstyledListItemThemeSettings"},"yiq":{"light":"#ffffff","dark":"#000000","__typename":"YiqThemeSettings"},"colorLightness":{"primaryDark":0.36,"primaryLight":0.74,"primaryLighter":0.89,"primaryLightest":0.95,"infoDark":0.39,"infoLight":0.72,"infoLighter":0.85,"infoLightest":0.93,"successDark":0.24,"successLight":0.62,"successLighter":0.8,"successLightest":0.91,"warningDark":0.39,"warningLight":0.68,"warningLighter":0.84,"warningLightest":0.93,"dangerDark":0.41,"dangerLight":0.72,"dangerLighter":0.89,"dangerLightest":0.95,"__typename":"ColorLightnessThemeSettings"},"localOverride":false,"__typename":"Theme"},"localOverride":false},"CachedAsset:text:en_US-components/common/EmailVerification-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/common/EmailVerification-1737571274000","value":{"email.verification.title":"Email Verification Required","email.verification.message.update.email":"To participate in the community, you must first verify your email address. The verification email was sent to {email}. To change your email, visit My Settings.","email.verification.message.resend.email":"To participate in the community, you must first verify your email address. The verification email was sent to {email}. Resend email."},"localOverride":false},"CachedAsset:text:en_US-shared/client/components/common/Loading/LoadingDot-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-shared/client/components/common/Loading/LoadingDot-1737571274000","value":{"title":"Loading..."},"localOverride":false},"CachedAsset:quilt:o365.prod:pages/blogs/BlogMessagePage:board:FastTrackforAzureBlog-1742487240076":{"__typename":"CachedAsset","id":"quilt:o365.prod:pages/blogs/BlogMessagePage:board:FastTrackforAzureBlog-1742487240076","value":{"id":"BlogMessagePage","container":{"id":"Common","headerProps":{"backgroundImageProps":null,"backgroundColor":null,"addComponents":null,"removeComponents":["community.widget.bannerWidget"],"componentOrder":null,"__typename":"QuiltContainerSectionProps"},"headerComponentProps":{"community.widget.breadcrumbWidget":{"disableLastCrumbForDesktop":false}},"footerProps":null,"footerComponentProps":null,"items":[{"id":"blog-article","layout":"ONE_COLUMN","bgColor":null,"showTitle":null,"showDescription":null,"textPosition":null,"textColor":null,"sectionEditLevel":"LOCKED","bgImage":null,"disableSpacing":null,"edgeToEdgeDisplay":null,"fullHeight":null,"showBorder":null,"__typename":"OneColumnQuiltSection","columnMap":{"main":[{"id":"blogs.widget.blogArticleWidget","className":"lia-blog-container","props":null,"__typename":"QuiltComponent"}],"__typename":"OneSectionColumns"}},{"id":"section-1729184836777","layout":"MAIN_SIDE","bgColor":"transparent","showTitle":false,"showDescription":false,"textPosition":"CENTER","textColor":"var(--lia-bs-body-color)","sectionEditLevel":null,"bgImage":null,"disableSpacing":null,"edgeToEdgeDisplay":null,"fullHeight":null,"showBorder":null,"__typename":"MainSideQuiltSection","columnMap":{"main":[],"side":[{"id":"custom.widget.Social_Sharing","className":null,"props":{"widgetVisibility":"signedInOrAnonymous","useTitle":true,"useBackground":true,"title":"Share","lazyLoad":false},"__typename":"QuiltComponent"}],"__typename":"MainSideSectionColumns"}}],"__typename":"QuiltContainer"},"__typename":"Quilt","localOverride":false},"localOverride":false},"CachedAsset:text:en_US-pages/blogs/BlogMessagePage-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-pages/blogs/BlogMessagePage-1737571274000","value":{"title":"{contextMessageSubject} | {communityTitle}","errorMissing":"This blog post cannot be found","name":"Blog Message Page","section.blog-article.title":"Blog Post","archivedMessageTitle":"This Content Has Been Archived","section.section-1729184836777.title":"","section.section-1729184836777.description":"","section.CncIde.title":"Blog Post","section.tifEmD.description":"","section.tifEmD.title":""},"localOverride":false},"CachedAsset:quiltWrapper:o365.prod:Common:1742487042167":{"__typename":"CachedAsset","id":"quiltWrapper:o365.prod:Common:1742487042167","value":{"id":"Common","header":{"backgroundImageProps":{"assetName":null,"backgroundSize":"COVER","backgroundRepeat":"NO_REPEAT","backgroundPosition":"CENTER_CENTER","lastModified":null,"__typename":"BackgroundImageProps"},"backgroundColor":"transparent","items":[{"id":"community.widget.navbarWidget","props":{"showUserName":true,"showRegisterLink":true,"useIconLanguagePicker":true,"useLabelLanguagePicker":true,"className":"QuiltComponent_lia-component-edit-mode__0nCcm","links":{"sideLinks":[],"mainLinks":[{"children":[],"linkType":"INTERNAL","id":"gxcuf89792","params":{},"routeName":"CommunityPage"},{"children":[],"linkType":"EXTERNAL","id":"external-link","url":"/Directory","target":"SELF"},{"children":[{"linkType":"INTERNAL","id":"microsoft365","params":{"categoryId":"microsoft365"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"microsoft-teams","params":{"categoryId":"MicrosoftTeams"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"windows","params":{"categoryId":"Windows"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"microsoft-securityand-compliance","params":{"categoryId":"microsoft-security"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"outlook","params":{"categoryId":"Outlook"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"planner","params":{"categoryId":"Planner"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"windows-server","params":{"categoryId":"Windows-Server"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"azure","params":{"categoryId":"Azure"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"exchange","params":{"categoryId":"Exchange"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"microsoft-endpoint-manager","params":{"categoryId":"microsoft-endpoint-manager"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"s-q-l-server","params":{"categoryId":"SQL-Server"},"routeName":"CategoryPage"},{"linkType":"EXTERNAL","id":"external-link-2","url":"/Directory","target":"SELF"}],"linkType":"EXTERNAL","id":"communities","url":"/","target":"BLANK"},{"children":[{"linkType":"INTERNAL","id":"education-sector","params":{"categoryId":"EducationSector"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"a-i","params":{"categoryId":"AI"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"i-t-ops-talk","params":{"categoryId":"ITOpsTalk"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"partner-community","params":{"categoryId":"PartnerCommunity"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"microsoft-mechanics","params":{"categoryId":"MicrosoftMechanics"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"healthcare-and-life-sciences","params":{"categoryId":"HealthcareAndLifeSciences"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"public-sector","params":{"categoryId":"PublicSector"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"io-t","params":{"categoryId":"IoT"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"driving-adoption","params":{"categoryId":"DrivingAdoption"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"s-m-b","params":{"categoryId":"SMB"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"startupsat-microsoft","params":{"categoryId":"StartupsatMicrosoft"},"routeName":"CategoryPage"},{"linkType":"EXTERNAL","id":"external-link-1","url":"/Directory","target":"SELF"}],"linkType":"EXTERNAL","id":"communities-1","url":"/","target":"SELF"},{"children":[],"linkType":"EXTERNAL","id":"external","url":"/Blogs","target":"SELF"},{"children":[],"linkType":"EXTERNAL","id":"external-1","url":"/Events","target":"SELF"},{"children":[{"linkType":"INTERNAL","id":"microsoft-learn-1","params":{"categoryId":"MicrosoftLearn"},"routeName":"CategoryPage"},{"linkType":"INTERNAL","id":"microsoft-learn-blog","params":{"boardId":"MicrosoftLearnBlog","categoryId":"MicrosoftLearn"},"routeName":"BlogBoardPage"},{"linkType":"EXTERNAL","id":"external-10","url":"https://learningroomdirectory.microsoft.com/","target":"BLANK"},{"linkType":"EXTERNAL","id":"external-3","url":"https://docs.microsoft.com/learn/dynamics365/?WT.mc_id=techcom_header-webpage-m365","target":"BLANK"},{"linkType":"EXTERNAL","id":"external-4","url":"https://docs.microsoft.com/learn/m365/?wt.mc_id=techcom_header-webpage-m365","target":"BLANK"},{"linkType":"EXTERNAL","id":"external-5","url":"https://docs.microsoft.com/learn/topics/sci/?wt.mc_id=techcom_header-webpage-m365","target":"BLANK"},{"linkType":"EXTERNAL","id":"external-6","url":"https://docs.microsoft.com/learn/powerplatform/?wt.mc_id=techcom_header-webpage-powerplatform","target":"BLANK"},{"linkType":"EXTERNAL","id":"external-7","url":"https://docs.microsoft.com/learn/github/?wt.mc_id=techcom_header-webpage-github","target":"BLANK"},{"linkType":"EXTERNAL","id":"external-8","url":"https://docs.microsoft.com/learn/teams/?wt.mc_id=techcom_header-webpage-teams","target":"BLANK"},{"linkType":"EXTERNAL","id":"external-9","url":"https://docs.microsoft.com/learn/dotnet/?wt.mc_id=techcom_header-webpage-dotnet","target":"BLANK"},{"linkType":"EXTERNAL","id":"external-2","url":"https://docs.microsoft.com/learn/azure/?WT.mc_id=techcom_header-webpage-m365","target":"BLANK"}],"linkType":"INTERNAL","id":"microsoft-learn","params":{"categoryId":"MicrosoftLearn"},"routeName":"CategoryPage"},{"children":[],"linkType":"INTERNAL","id":"community-info-center","params":{"categoryId":"Community-Info-Center"},"routeName":"CategoryPage"}]},"style":{"boxShadow":"var(--lia-bs-box-shadow-sm)","controllerHighlightColor":"hsla(30, 100%, 50%)","linkFontWeight":"400","dropdownDividerMarginBottom":"10px","hamburgerBorderHover":"none","linkBoxShadowHover":"none","linkFontSize":"14px","backgroundOpacity":0.8,"controllerBorderRadius":"var(--lia-border-radius-50)","hamburgerBgColor":"transparent","hamburgerColor":"var(--lia-nav-controller-icon-color)","linkTextBorderBottom":"none","brandLogoHeight":"30px","linkBgHoverColor":"transparent","linkLetterSpacing":"normal","collapseMenuDividerOpacity":0.16,"dropdownPaddingBottom":"15px","paddingBottom":"15px","dropdownMenuOffset":"2px","hamburgerBgHoverColor":"transparent","borderBottom":"1px solid var(--lia-bs-border-color)","hamburgerBorder":"none","dropdownPaddingX":"10px","brandMarginRightSm":"10px","linkBoxShadow":"none","collapseMenuDividerBg":"var(--lia-nav-link-color)","linkColor":"var(--lia-bs-body-color)","linkJustifyContent":"flex-start","dropdownPaddingTop":"10px","controllerHighlightTextColor":"var(--lia-yiq-dark)","controllerTextColor":"var(--lia-nav-controller-icon-color)","background":{"imageAssetName":"","color":"var(--lia-bs-white)","size":"COVER","repeat":"NO_REPEAT","position":"CENTER_CENTER","imageLastModified":""},"linkBorderRadius":"var(--lia-bs-border-radius-sm)","linkHoverColor":"var(--lia-bs-body-color)","position":"FIXED","linkBorder":"none","linkTextBorderBottomHover":"2px solid var(--lia-bs-body-color)","brandMarginRight":"30px","hamburgerHoverColor":"var(--lia-nav-controller-icon-color)","linkBorderHover":"none","collapseMenuMarginLeft":"20px","linkFontStyle":"NORMAL","controllerTextHoverColor":"var(--lia-nav-controller-icon-hover-color)","linkPaddingX":"10px","linkPaddingY":"5px","paddingTop":"15px","linkTextTransform":"NONE","dropdownBorderColor":"hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.08)","controllerBgHoverColor":"hsla(var(--lia-bs-black-h), var(--lia-bs-black-s), var(--lia-bs-black-l), 0.1)","linkBgColor":"transparent","linkDropdownPaddingX":"var(--lia-nav-link-px)","linkDropdownPaddingY":"9px","controllerIconColor":"var(--lia-bs-body-color)","dropdownDividerMarginTop":"10px","linkGap":"10px","controllerIconHoverColor":"var(--lia-bs-body-color)"},"showSearchIcon":false,"languagePickerStyle":"iconAndLabel"},"__typename":"QuiltComponent"},{"id":"community.widget.breadcrumbWidget","props":{"backgroundColor":"transparent","linkHighlightColor":"var(--lia-bs-primary)","visualEffects":{"showBottomBorder":true},"linkTextColor":"var(--lia-bs-gray-700)"},"__typename":"QuiltComponent"},{"id":"custom.widget.community_banner","props":{"widgetVisibility":"signedInOrAnonymous","useTitle":true,"usePageWidth":false,"useBackground":false,"title":"","lazyLoad":false},"__typename":"QuiltComponent"},{"id":"custom.widget.HeroBanner","props":{"widgetVisibility":"signedInOrAnonymous","usePageWidth":false,"useTitle":true,"cMax_items":3,"useBackground":false,"title":"","lazyLoad":false,"widgetChooser":"custom.widget.HeroBanner"},"__typename":"QuiltComponent"}],"__typename":"QuiltWrapperSection"},"footer":{"backgroundImageProps":{"assetName":null,"backgroundSize":"COVER","backgroundRepeat":"NO_REPEAT","backgroundPosition":"CENTER_CENTER","lastModified":null,"__typename":"BackgroundImageProps"},"backgroundColor":"transparent","items":[{"id":"custom.widget.MicrosoftFooter","props":{"widgetVisibility":"signedInOrAnonymous","useTitle":true,"useBackground":false,"title":"","lazyLoad":false},"__typename":"QuiltComponent"}],"__typename":"QuiltWrapperSection"},"__typename":"QuiltWrapper","localOverride":false},"localOverride":false},"CachedAsset:text:en_US-components/common/ActionFeedback-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/common/ActionFeedback-1737571274000","value":{"joinedGroupHub.title":"Welcome","joinedGroupHub.message":"You are now a member of this group and are subscribed to updates.","groupHubInviteNotFound.title":"Invitation Not Found","groupHubInviteNotFound.message":"Sorry, we could not find your invitation to the group. The owner may have canceled the invite.","groupHubNotFound.title":"Group Not Found","groupHubNotFound.message":"The grouphub you tried to join does not exist. It may have been deleted.","existingGroupHubMember.title":"Already Joined","existingGroupHubMember.message":"You are already a member of this group.","accountLocked.title":"Account Locked","accountLocked.message":"Your account has been locked due to multiple failed attempts. Try again in {lockoutTime} minutes.","editedGroupHub.title":"Changes Saved","editedGroupHub.message":"Your group has been updated.","leftGroupHub.title":"Goodbye","leftGroupHub.message":"You are no longer a member of this group and will not receive future updates.","deletedGroupHub.title":"Deleted","deletedGroupHub.message":"The group has been deleted.","groupHubCreated.title":"Group Created","groupHubCreated.message":"{groupHubName} is ready to use","accountClosed.title":"Account Closed","accountClosed.message":"The account has been closed and you will now be redirected to the homepage","resetTokenExpired.title":"Reset Password Link has Expired","resetTokenExpired.message":"Try resetting your password again","invalidUrl.title":"Invalid URL","invalidUrl.message":"The URL you're using is not recognized. Verify your URL and try again.","accountClosedForUser.title":"Account Closed","accountClosedForUser.message":"{userName}'s account is closed","inviteTokenInvalid.title":"Invitation Invalid","inviteTokenInvalid.message":"Your invitation to the community has been canceled or expired.","inviteTokenError.title":"Invitation Verification Failed","inviteTokenError.message":"The url you are utilizing is not recognized. Verify your URL and try again","pageNotFound.title":"Access Denied","pageNotFound.message":"You do not have access to this area of the community or it doesn't exist","eventAttending.title":"Responded as Attending","eventAttending.message":"You'll be notified when there's new activity and reminded as the event approaches","eventInterested.title":"Responded as Interested","eventInterested.message":"You'll be notified when there's new activity and reminded as the event approaches","eventNotFound.title":"Event Not Found","eventNotFound.message":"The event you tried to respond to does not exist.","redirectToRelatedPage.title":"Showing Related Content","redirectToRelatedPageForBaseUsers.title":"Showing Related Content","redirectToRelatedPageForBaseUsers.message":"The content you are trying to access is archived","redirectToRelatedPage.message":"The content you are trying to access is archived","relatedUrl.archivalLink.flyoutMessage":"The content you are trying to access is archived View Archived Content"},"localOverride":false},"CachedAsset:component:custom.widget.community_banner-en-1742487283414":{"__typename":"CachedAsset","id":"component:custom.widget.community_banner-en-1742487283414","value":{"component":{"id":"custom.widget.community_banner","template":{"id":"community_banner","markupLanguage":"HANDLEBARS","style":".community-banner {\n a.top-bar.btn {\n top: 0px;\n width: 100%;\n z-index: 999;\n text-align: center;\n left: 0px;\n background: #0068b8;\n color: white;\n padding: 10px 0px;\n display:block;\n box-shadow:none !important;\n border: none !important;\n border-radius: none !important;\n margin: 0px !important;\n font-size:14px;\n }\n}","texts":null,"defaults":{"config":{"applicablePages":[],"description":"community announcement text","fetchedContent":null,"__typename":"ComponentConfiguration"},"props":[],"__typename":"ComponentProperties"},"components":[{"id":"custom.widget.community_banner","form":null,"config":null,"props":[],"__typename":"Component"}],"grouping":"CUSTOM","__typename":"ComponentTemplate"},"properties":{"config":{"applicablePages":[],"description":"community announcement text","fetchedContent":null,"__typename":"ComponentConfiguration"},"props":[],"__typename":"ComponentProperties"},"form":null,"__typename":"Component","localOverride":false},"globalCss":{"css":".custom_widget_community_banner_community-banner_1a5zb_1 {\n a.custom_widget_community_banner_top-bar_1a5zb_2.custom_widget_community_banner_btn_1a5zb_2 {\n top: 0;\n width: 100%;\n z-index: 999;\n text-align: center;\n left: 0;\n background: #0068b8;\n color: white;\n padding: 0.625rem 0;\n display:block;\n box-shadow:none !important;\n border: none !important;\n border-radius: none !important;\n margin: 0 !important;\n font-size:0.875rem;\n }\n}","tokens":{"community-banner":"custom_widget_community_banner_community-banner_1a5zb_1","top-bar":"custom_widget_community_banner_top-bar_1a5zb_2","btn":"custom_widget_community_banner_btn_1a5zb_2"}},"form":null},"localOverride":false},"CachedAsset:component:custom.widget.HeroBanner-en-1742487283414":{"__typename":"CachedAsset","id":"component:custom.widget.HeroBanner-en-1742487283414","value":{"component":{"id":"custom.widget.HeroBanner","template":{"id":"HeroBanner","markupLanguage":"REACT","style":null,"texts":{"searchPlaceholderText":"Search this community","followActionText":"Follow","unfollowActionText":"Following","searchOnHoverText":"Please enter your search term(s) and then press return key to complete a search."},"defaults":{"config":{"applicablePages":[],"description":null,"fetchedContent":null,"__typename":"ComponentConfiguration"},"props":[{"id":"max_items","dataType":"NUMBER","list":false,"defaultValue":"3","label":"Max Items","description":"The maximum number of items to display in the carousel","possibleValues":null,"control":"INPUT","__typename":"PropDefinition"}],"__typename":"ComponentProperties"},"components":[{"id":"custom.widget.HeroBanner","form":{"fields":[{"id":"widgetChooser","validation":null,"noValidation":null,"dataType":"STRING","list":null,"control":null,"defaultValue":null,"label":null,"description":null,"possibleValues":null,"__typename":"FormField"},{"id":"title","validation":null,"noValidation":null,"dataType":"STRING","list":null,"control":null,"defaultValue":null,"label":null,"description":null,"possibleValues":null,"__typename":"FormField"},{"id":"useTitle","validation":null,"noValidation":null,"dataType":"BOOLEAN","list":null,"control":null,"defaultValue":null,"label":null,"description":null,"possibleValues":null,"__typename":"FormField"},{"id":"useBackground","validation":null,"noValidation":null,"dataType":"BOOLEAN","list":null,"control":null,"defaultValue":null,"label":null,"description":null,"possibleValues":null,"__typename":"FormField"},{"id":"widgetVisibility","validation":null,"noValidation":null,"dataType":"STRING","list":null,"control":null,"defaultValue":null,"label":null,"description":null,"possibleValues":null,"__typename":"FormField"},{"id":"moreOptions","validation":null,"noValidation":null,"dataType":"STRING","list":null,"control":null,"defaultValue":null,"label":null,"description":null,"possibleValues":null,"__typename":"FormField"},{"id":"cMax_items","validation":null,"noValidation":null,"dataType":"NUMBER","list":false,"control":"INPUT","defaultValue":"3","label":"Max Items","description":"The maximum number of items to display in the carousel","possibleValues":null,"__typename":"FormField"}],"layout":{"rows":[{"id":"widgetChooserGroup","type":"fieldset","as":null,"items":[{"id":"widgetChooser","className":null,"__typename":"FormFieldRef"}],"props":null,"legend":null,"description":null,"className":null,"viewVariant":null,"toggleState":null,"__typename":"FormFieldset"},{"id":"titleGroup","type":"fieldset","as":null,"items":[{"id":"title","className":null,"__typename":"FormFieldRef"},{"id":"useTitle","className":null,"__typename":"FormFieldRef"}],"props":null,"legend":null,"description":null,"className":null,"viewVariant":null,"toggleState":null,"__typename":"FormFieldset"},{"id":"useBackground","type":"fieldset","as":null,"items":[{"id":"useBackground","className":null,"__typename":"FormFieldRef"}],"props":null,"legend":null,"description":null,"className":null,"viewVariant":null,"toggleState":null,"__typename":"FormFieldset"},{"id":"widgetVisibility","type":"fieldset","as":null,"items":[{"id":"widgetVisibility","className":null,"__typename":"FormFieldRef"}],"props":null,"legend":null,"description":null,"className":null,"viewVariant":null,"toggleState":null,"__typename":"FormFieldset"},{"id":"moreOptionsGroup","type":"fieldset","as":null,"items":[{"id":"moreOptions","className":null,"__typename":"FormFieldRef"}],"props":null,"legend":null,"description":null,"className":null,"viewVariant":null,"toggleState":null,"__typename":"FormFieldset"},{"id":"componentPropsGroup","type":"fieldset","as":null,"items":[{"id":"cMax_items","className":null,"__typename":"FormFieldRef"}],"props":null,"legend":null,"description":null,"className":null,"viewVariant":null,"toggleState":null,"__typename":"FormFieldset"}],"actionButtons":null,"className":"custom_widget_HeroBanner_form","formGroupFieldSeparator":"divider","__typename":"FormLayout"},"__typename":"Form"},"config":null,"props":[],"__typename":"Component"}],"grouping":"CUSTOM","__typename":"ComponentTemplate"},"properties":{"config":{"applicablePages":[],"description":null,"fetchedContent":null,"__typename":"ComponentConfiguration"},"props":[{"id":"max_items","dataType":"NUMBER","list":false,"defaultValue":"3","label":"Max Items","description":"The maximum number of items to display in the carousel","possibleValues":null,"control":"INPUT","__typename":"PropDefinition"}],"__typename":"ComponentProperties"},"form":{"fields":[{"id":"widgetChooser","validation":null,"noValidation":null,"dataType":"STRING","list":null,"control":null,"defaultValue":null,"label":null,"description":null,"possibleValues":null,"__typename":"FormField"},{"id":"title","validation":null,"noValidation":null,"dataType":"STRING","list":null,"control":null,"defaultValue":null,"label":null,"description":null,"possibleValues":null,"__typename":"FormField"},{"id":"useTitle","validation":null,"noValidation":null,"dataType":"BOOLEAN","list":null,"control":null,"defaultValue":null,"label":null,"description":null,"possibleValues":null,"__typename":"FormField"},{"id":"useBackground","validation":null,"noValidation":null,"dataType":"BOOLEAN","list":null,"control":null,"defaultValue":null,"label":null,"description":null,"possibleValues":null,"__typename":"FormField"},{"id":"widgetVisibility","validation":null,"noValidation":null,"dataType":"STRING","list":null,"control":null,"defaultValue":null,"label":null,"description":null,"possibleValues":null,"__typename":"FormField"},{"id":"moreOptions","validation":null,"noValidation":null,"dataType":"STRING","list":null,"control":null,"defaultValue":null,"label":null,"description":null,"possibleValues":null,"__typename":"FormField"},{"id":"cMax_items","validation":null,"noValidation":null,"dataType":"NUMBER","list":false,"control":"INPUT","defaultValue":"3","label":"Max Items","description":"The maximum number of items to display in the carousel","possibleValues":null,"__typename":"FormField"}],"layout":{"rows":[{"id":"widgetChooserGroup","type":"fieldset","as":null,"items":[{"id":"widgetChooser","className":null,"__typename":"FormFieldRef"}],"props":null,"legend":null,"description":null,"className":null,"viewVariant":null,"toggleState":null,"__typename":"FormFieldset"},{"id":"titleGroup","type":"fieldset","as":null,"items":[{"id":"title","className":null,"__typename":"FormFieldRef"},{"id":"useTitle","className":null,"__typename":"FormFieldRef"}],"props":null,"legend":null,"description":null,"className":null,"viewVariant":null,"toggleState":null,"__typename":"FormFieldset"},{"id":"useBackground","type":"fieldset","as":null,"items":[{"id":"useBackground","className":null,"__typename":"FormFieldRef"}],"props":null,"legend":null,"description":null,"className":null,"viewVariant":null,"toggleState":null,"__typename":"FormFieldset"},{"id":"widgetVisibility","type":"fieldset","as":null,"items":[{"id":"widgetVisibility","className":null,"__typename":"FormFieldRef"}],"props":null,"legend":null,"description":null,"className":null,"viewVariant":null,"toggleState":null,"__typename":"FormFieldset"},{"id":"moreOptionsGroup","type":"fieldset","as":null,"items":[{"id":"moreOptions","className":null,"__typename":"FormFieldRef"}],"props":null,"legend":null,"description":null,"className":null,"viewVariant":null,"toggleState":null,"__typename":"FormFieldset"},{"id":"componentPropsGroup","type":"fieldset","as":null,"items":[{"id":"cMax_items","className":null,"__typename":"FormFieldRef"}],"props":null,"legend":null,"description":null,"className":null,"viewVariant":null,"toggleState":null,"__typename":"FormFieldset"}],"actionButtons":null,"className":"custom_widget_HeroBanner_form","formGroupFieldSeparator":"divider","__typename":"FormLayout"},"__typename":"Form"},"__typename":"Component","localOverride":false},"globalCss":null,"form":{"fields":[{"id":"widgetChooser","validation":null,"noValidation":null,"dataType":"STRING","list":null,"control":null,"defaultValue":null,"label":null,"description":null,"possibleValues":null,"__typename":"FormField"},{"id":"title","validation":null,"noValidation":null,"dataType":"STRING","list":null,"control":null,"defaultValue":null,"label":null,"description":null,"possibleValues":null,"__typename":"FormField"},{"id":"useTitle","validation":null,"noValidation":null,"dataType":"BOOLEAN","list":null,"control":null,"defaultValue":null,"label":null,"description":null,"possibleValues":null,"__typename":"FormField"},{"id":"useBackground","validation":null,"noValidation":null,"dataType":"BOOLEAN","list":null,"control":null,"defaultValue":null,"label":null,"description":null,"possibleValues":null,"__typename":"FormField"},{"id":"widgetVisibility","validation":null,"noValidation":null,"dataType":"STRING","list":null,"control":null,"defaultValue":null,"label":null,"description":null,"possibleValues":null,"__typename":"FormField"},{"id":"moreOptions","validation":null,"noValidation":null,"dataType":"STRING","list":null,"control":null,"defaultValue":null,"label":null,"description":null,"possibleValues":null,"__typename":"FormField"},{"id":"cMax_items","validation":null,"noValidation":null,"dataType":"NUMBER","list":false,"control":"INPUT","defaultValue":"3","label":"Max Items","description":"The maximum number of items to display in the carousel","possibleValues":null,"__typename":"FormField"}],"layout":{"rows":[{"id":"widgetChooserGroup","type":"fieldset","as":null,"items":[{"id":"widgetChooser","className":null,"__typename":"FormFieldRef"}],"props":null,"legend":null,"description":null,"className":null,"viewVariant":null,"toggleState":null,"__typename":"FormFieldset"},{"id":"titleGroup","type":"fieldset","as":null,"items":[{"id":"title","className":null,"__typename":"FormFieldRef"},{"id":"useTitle","className":null,"__typename":"FormFieldRef"}],"props":null,"legend":null,"description":null,"className":null,"viewVariant":null,"toggleState":null,"__typename":"FormFieldset"},{"id":"useBackground","type":"fieldset","as":null,"items":[{"id":"useBackground","className":null,"__typename":"FormFieldRef"}],"props":null,"legend":null,"description":null,"className":null,"viewVariant":null,"toggleState":null,"__typename":"FormFieldset"},{"id":"widgetVisibility","type":"fieldset","as":null,"items":[{"id":"widgetVisibility","className":null,"__typename":"FormFieldRef"}],"props":null,"legend":null,"description":null,"className":null,"viewVariant":null,"toggleState":null,"__typename":"FormFieldset"},{"id":"moreOptionsGroup","type":"fieldset","as":null,"items":[{"id":"moreOptions","className":null,"__typename":"FormFieldRef"}],"props":null,"legend":null,"description":null,"className":null,"viewVariant":null,"toggleState":null,"__typename":"FormFieldset"},{"id":"componentPropsGroup","type":"fieldset","as":null,"items":[{"id":"cMax_items","className":null,"__typename":"FormFieldRef"}],"props":null,"legend":null,"description":null,"className":null,"viewVariant":null,"toggleState":null,"__typename":"FormFieldset"}],"actionButtons":null,"className":"custom_widget_HeroBanner_form","formGroupFieldSeparator":"divider","__typename":"FormLayout"},"__typename":"Form"}},"localOverride":false},"CachedAsset:component:custom.widget.Social_Sharing-en-1742487283414":{"__typename":"CachedAsset","id":"component:custom.widget.Social_Sharing-en-1742487283414","value":{"component":{"id":"custom.widget.Social_Sharing","template":{"id":"Social_Sharing","markupLanguage":"HANDLEBARS","style":".social-share {\n .sharing-options {\n position: relative;\n margin: 0;\n padding: 0;\n line-height: 10px;\n display: flex;\n justify-content: left;\n gap: 5px;\n list-style-type: none;\n li {\n text-align: left;\n a {\n min-width: 30px;\n min-height: 30px;\n display: block;\n padding: 1px;\n .social-share-linkedin {\n img {\n background-color: rgb(0, 119, 181);\n }\n }\n .social-share-facebook {\n img {\n background-color: rgb(59, 89, 152);\n }\n }\n .social-share-x {\n img {\n background-color: rgb(0, 0, 0);\n }\n }\n .social-share-rss {\n img {\n background-color: rgb(0, 0, 0);\n }\n }\n .social-share-reddit {\n img {\n background-color: rgb(255, 69, 0);\n }\n }\n .social-share-email {\n img {\n background-color: rgb(132, 132, 132);\n }\n }\n }\n a {\n img {\n height: 2rem;\n }\n }\n }\n }\n}\n","texts":null,"defaults":{"config":{"applicablePages":[],"description":"Adds buttons to share to various social media websites","fetchedContent":null,"__typename":"ComponentConfiguration"},"props":[],"__typename":"ComponentProperties"},"components":[{"id":"custom.widget.Social_Sharing","form":null,"config":null,"props":[],"__typename":"Component"}],"grouping":"CUSTOM","__typename":"ComponentTemplate"},"properties":{"config":{"applicablePages":[],"description":"Adds buttons to share to various social media websites","fetchedContent":null,"__typename":"ComponentConfiguration"},"props":[],"__typename":"ComponentProperties"},"form":null,"__typename":"Component","localOverride":false},"globalCss":{"css":".custom_widget_Social_Sharing_social-share_c7xxz_1 {\n .custom_widget_Social_Sharing_sharing-options_c7xxz_2 {\n position: relative;\n margin: 0;\n padding: 0;\n line-height: 0.625rem;\n display: flex;\n justify-content: left;\n gap: 0.3125rem;\n list-style-type: none;\n li {\n text-align: left;\n a {\n min-width: 1.875rem;\n min-height: 1.875rem;\n display: block;\n padding: 0.0625rem;\n .custom_widget_Social_Sharing_social-share-linkedin_c7xxz_18 {\n img {\n background-color: rgb(0, 119, 181);\n }\n }\n .custom_widget_Social_Sharing_social-share-facebook_c7xxz_23 {\n img {\n background-color: rgb(59, 89, 152);\n }\n }\n .custom_widget_Social_Sharing_social-share-x_c7xxz_28 {\n img {\n background-color: rgb(0, 0, 0);\n }\n }\n .custom_widget_Social_Sharing_social-share-rss_c7xxz_33 {\n img {\n background-color: rgb(0, 0, 0);\n }\n }\n .custom_widget_Social_Sharing_social-share-reddit_c7xxz_38 {\n img {\n background-color: rgb(255, 69, 0);\n }\n }\n .custom_widget_Social_Sharing_social-share-email_c7xxz_43 {\n img {\n background-color: rgb(132, 132, 132);\n }\n }\n }\n a {\n img {\n height: 2rem;\n }\n }\n }\n }\n}\n","tokens":{"social-share":"custom_widget_Social_Sharing_social-share_c7xxz_1","sharing-options":"custom_widget_Social_Sharing_sharing-options_c7xxz_2","social-share-linkedin":"custom_widget_Social_Sharing_social-share-linkedin_c7xxz_18","social-share-facebook":"custom_widget_Social_Sharing_social-share-facebook_c7xxz_23","social-share-x":"custom_widget_Social_Sharing_social-share-x_c7xxz_28","social-share-rss":"custom_widget_Social_Sharing_social-share-rss_c7xxz_33","social-share-reddit":"custom_widget_Social_Sharing_social-share-reddit_c7xxz_38","social-share-email":"custom_widget_Social_Sharing_social-share-email_c7xxz_43"}},"form":null},"localOverride":false},"CachedAsset:component:custom.widget.MicrosoftFooter-en-1742487283414":{"__typename":"CachedAsset","id":"component:custom.widget.MicrosoftFooter-en-1742487283414","value":{"component":{"id":"custom.widget.MicrosoftFooter","template":{"id":"MicrosoftFooter","markupLanguage":"HANDLEBARS","style":".context-uhf {\n min-width: 280px;\n font-size: 15px;\n box-sizing: border-box;\n -ms-text-size-adjust: 100%;\n -webkit-text-size-adjust: 100%;\n & *,\n & *:before,\n & *:after {\n box-sizing: inherit;\n }\n a.c-uhff-link {\n color: #616161;\n word-break: break-word;\n text-decoration: none;\n }\n &a:link,\n &a:focus,\n &a:hover,\n &a:active,\n &a:visited {\n text-decoration: none;\n color: inherit;\n }\n & div {\n font-family: 'Segoe UI', SegoeUI, 'Helvetica Neue', Helvetica, Arial, sans-serif;\n }\n}\n.c-uhff {\n background: #f2f2f2;\n margin: -1.5625;\n width: auto;\n height: auto;\n}\n.c-uhff-nav {\n margin: 0 auto;\n max-width: calc(1600px + 10%);\n padding: 0 5%;\n box-sizing: inherit;\n &:before,\n &:after {\n content: ' ';\n display: table;\n clear: left;\n }\n @media only screen and (max-width: 1083px) {\n padding-left: 12px;\n }\n .c-heading-4 {\n color: #616161;\n word-break: break-word;\n font-size: 15px;\n line-height: 20px;\n padding: 36px 0 4px;\n font-weight: 600;\n }\n .c-uhff-nav-row {\n .c-uhff-nav-group {\n display: block;\n float: left;\n min-height: 1px;\n vertical-align: text-top;\n padding: 0 12px;\n width: 100%;\n zoom: 1;\n &:first-child {\n padding-left: 0;\n @media only screen and (max-width: 1083px) {\n padding-left: 12px;\n }\n }\n @media only screen and (min-width: 540px) and (max-width: 1082px) {\n width: 33.33333%;\n }\n @media only screen and (min-width: 1083px) {\n width: 16.6666666667%;\n }\n ul.c-list.f-bare {\n font-size: 11px;\n line-height: 16px;\n margin-top: 0;\n margin-bottom: 0;\n padding-left: 0;\n list-style-type: none;\n li {\n word-break: break-word;\n padding: 8px 0;\n margin: 0;\n }\n }\n }\n }\n}\n.c-uhff-base {\n background: #f2f2f2;\n margin: 0 auto;\n max-width: calc(1600px + 10%);\n padding: 30px 5% 16px;\n &:before,\n &:after {\n content: ' ';\n display: table;\n }\n &:after {\n clear: both;\n }\n a.c-uhff-ccpa {\n font-size: 11px;\n line-height: 16px;\n float: left;\n margin: 3px 0;\n }\n a.c-uhff-ccpa:hover {\n text-decoration: underline;\n }\n ul.c-list {\n font-size: 11px;\n line-height: 16px;\n float: right;\n margin: 3px 0;\n color: #616161;\n li {\n padding: 0 24px 4px 0;\n display: inline-block;\n }\n }\n .c-list.f-bare {\n padding-left: 0;\n list-style-type: none;\n }\n @media only screen and (max-width: 1083px) {\n display: flex;\n flex-wrap: wrap;\n padding: 30px 24px 16px;\n }\n}\n","texts":{"New tab":"What's New","New 1":"Surface Laptop Studio 2","New 2":"Surface Laptop Go 3","New 3":"Surface Pro 9","New 4":"Surface Laptop 5","New 5":"Surface Studio 2+","New 6":"Copilot in Windows","New 7":"Microsoft 365","New 8":"Windows 11 apps","Store tab":"Microsoft Store","Store 1":"Account Profile","Store 2":"Download Center","Store 3":"Microsoft Store Support","Store 4":"Returns","Store 5":"Order tracking","Store 6":"Certified Refurbished","Store 7":"Microsoft Store Promise","Store 8":"Flexible Payments","Education tab":"Education","Edu 1":"Microsoft in education","Edu 2":"Devices for education","Edu 3":"Microsoft Teams for Education","Edu 4":"Microsoft 365 Education","Edu 5":"How to buy for your school","Edu 6":"Educator Training and development","Edu 7":"Deals for students and parents","Edu 8":"Azure for students","Business tab":"Business","Bus 1":"Microsoft Cloud","Bus 2":"Microsoft Security","Bus 3":"Dynamics 365","Bus 4":"Microsoft 365","Bus 5":"Microsoft Power Platform","Bus 6":"Microsoft Teams","Bus 7":"Microsoft Industry","Bus 8":"Small Business","Developer tab":"Developer & IT","Dev 1":"Azure","Dev 2":"Developer Center","Dev 3":"Documentation","Dev 4":"Microsoft Learn","Dev 5":"Microsoft Tech Community","Dev 6":"Azure Marketplace","Dev 7":"AppSource","Dev 8":"Visual Studio","Company tab":"Company","Com 1":"Careers","Com 2":"About Microsoft","Com 3":"Company News","Com 4":"Privacy at Microsoft","Com 5":"Investors","Com 6":"Diversity and inclusion","Com 7":"Accessiblity","Com 8":"Sustainibility"},"defaults":{"config":{"applicablePages":[],"description":"The Microsoft Footer","fetchedContent":null,"__typename":"ComponentConfiguration"},"props":[],"__typename":"ComponentProperties"},"components":[{"id":"custom.widget.MicrosoftFooter","form":null,"config":null,"props":[],"__typename":"Component"}],"grouping":"CUSTOM","__typename":"ComponentTemplate"},"properties":{"config":{"applicablePages":[],"description":"The Microsoft Footer","fetchedContent":null,"__typename":"ComponentConfiguration"},"props":[],"__typename":"ComponentProperties"},"form":null,"__typename":"Component","localOverride":false},"globalCss":{"css":".custom_widget_MicrosoftFooter_context-uhf_f95yq_1 {\n min-width: 17.5rem;\n font-size: 0.9375rem;\n box-sizing: border-box;\n -ms-text-size-adjust: 100%;\n -webkit-text-size-adjust: 100%;\n & *,\n & *:before,\n & *:after {\n box-sizing: inherit;\n }\n a.custom_widget_MicrosoftFooter_c-uhff-link_f95yq_12 {\n color: #616161;\n word-break: break-word;\n text-decoration: none;\n }\n &a:link,\n &a:focus,\n &a:hover,\n &a:active,\n &a:visited {\n text-decoration: none;\n color: inherit;\n }\n & div {\n font-family: 'Segoe UI', SegoeUI, 'Helvetica Neue', Helvetica, Arial, sans-serif;\n }\n}\n.custom_widget_MicrosoftFooter_c-uhff_f95yq_12 {\n background: #f2f2f2;\n margin: -1.5625;\n width: auto;\n height: auto;\n}\n.custom_widget_MicrosoftFooter_c-uhff-nav_f95yq_35 {\n margin: 0 auto;\n max-width: calc(100rem + 10%);\n padding: 0 5%;\n box-sizing: inherit;\n &:before,\n &:after {\n content: ' ';\n display: table;\n clear: left;\n }\n @media only screen and (max-width: 1083px) {\n padding-left: 0.75rem;\n }\n .custom_widget_MicrosoftFooter_c-heading-4_f95yq_49 {\n color: #616161;\n word-break: break-word;\n font-size: 0.9375rem;\n line-height: 1.25rem;\n padding: 2.25rem 0 0.25rem;\n font-weight: 600;\n }\n .custom_widget_MicrosoftFooter_c-uhff-nav-row_f95yq_57 {\n .custom_widget_MicrosoftFooter_c-uhff-nav-group_f95yq_58 {\n display: block;\n float: left;\n min-height: 0.0625rem;\n vertical-align: text-top;\n padding: 0 0.75rem;\n width: 100%;\n zoom: 1;\n &:first-child {\n padding-left: 0;\n @media only screen and (max-width: 1083px) {\n padding-left: 0.75rem;\n }\n }\n @media only screen and (min-width: 540px) and (max-width: 1082px) {\n width: 33.33333%;\n }\n @media only screen and (min-width: 1083px) {\n width: 16.6666666667%;\n }\n ul.custom_widget_MicrosoftFooter_c-list_f95yq_78.custom_widget_MicrosoftFooter_f-bare_f95yq_78 {\n font-size: 0.6875rem;\n line-height: 1rem;\n margin-top: 0;\n margin-bottom: 0;\n padding-left: 0;\n list-style-type: none;\n li {\n word-break: break-word;\n padding: 0.5rem 0;\n margin: 0;\n }\n }\n }\n }\n}\n.custom_widget_MicrosoftFooter_c-uhff-base_f95yq_94 {\n background: #f2f2f2;\n margin: 0 auto;\n max-width: calc(100rem + 10%);\n padding: 1.875rem 5% 1rem;\n &:before,\n &:after {\n content: ' ';\n display: table;\n }\n &:after {\n clear: both;\n }\n a.custom_widget_MicrosoftFooter_c-uhff-ccpa_f95yq_107 {\n font-size: 0.6875rem;\n line-height: 1rem;\n float: left;\n margin: 0.1875rem 0;\n }\n a.custom_widget_MicrosoftFooter_c-uhff-ccpa_f95yq_107:hover {\n text-decoration: underline;\n }\n ul.custom_widget_MicrosoftFooter_c-list_f95yq_78 {\n font-size: 0.6875rem;\n line-height: 1rem;\n float: right;\n margin: 0.1875rem 0;\n color: #616161;\n li {\n padding: 0 1.5rem 0.25rem 0;\n display: inline-block;\n }\n }\n .custom_widget_MicrosoftFooter_c-list_f95yq_78.custom_widget_MicrosoftFooter_f-bare_f95yq_78 {\n padding-left: 0;\n list-style-type: none;\n }\n @media only screen and (max-width: 1083px) {\n display: flex;\n flex-wrap: wrap;\n padding: 1.875rem 1.5rem 1rem;\n }\n}\n","tokens":{"context-uhf":"custom_widget_MicrosoftFooter_context-uhf_f95yq_1","c-uhff-link":"custom_widget_MicrosoftFooter_c-uhff-link_f95yq_12","c-uhff":"custom_widget_MicrosoftFooter_c-uhff_f95yq_12","c-uhff-nav":"custom_widget_MicrosoftFooter_c-uhff-nav_f95yq_35","c-heading-4":"custom_widget_MicrosoftFooter_c-heading-4_f95yq_49","c-uhff-nav-row":"custom_widget_MicrosoftFooter_c-uhff-nav-row_f95yq_57","c-uhff-nav-group":"custom_widget_MicrosoftFooter_c-uhff-nav-group_f95yq_58","c-list":"custom_widget_MicrosoftFooter_c-list_f95yq_78","f-bare":"custom_widget_MicrosoftFooter_f-bare_f95yq_78","c-uhff-base":"custom_widget_MicrosoftFooter_c-uhff-base_f95yq_94","c-uhff-ccpa":"custom_widget_MicrosoftFooter_c-uhff-ccpa_f95yq_107"}},"form":null},"localOverride":false},"CachedAsset:text:en_US-components/community/Breadcrumb-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/community/Breadcrumb-1737571274000","value":{"navLabel":"Breadcrumbs","dropdown":"Additional parent page navigation"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageBanner-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageBanner-1737571274000","value":{"messageMarkedAsSpam":"This post has been marked as spam","messageMarkedAsSpam@board:TKB":"This article has been marked as spam","messageMarkedAsSpam@board:BLOG":"This post has been marked as spam","messageMarkedAsSpam@board:FORUM":"This discussion has been marked as spam","messageMarkedAsSpam@board:OCCASION":"This event has been marked as spam","messageMarkedAsSpam@board:IDEA":"This idea has been marked as spam","manageSpam":"Manage Spam","messageMarkedAsAbuse":"This post has been marked as abuse","messageMarkedAsAbuse@board:TKB":"This article has been marked as abuse","messageMarkedAsAbuse@board:BLOG":"This post has been marked as abuse","messageMarkedAsAbuse@board:FORUM":"This discussion has been marked as abuse","messageMarkedAsAbuse@board:OCCASION":"This event has been marked as abuse","messageMarkedAsAbuse@board:IDEA":"This idea has been marked as abuse","preModCommentAuthorText":"This comment will be published as soon as it is approved","preModCommentModeratorText":"This comment is awaiting moderation","messageMarkedAsOther":"This post has been rejected due to other reasons","messageMarkedAsOther@board:TKB":"This article has been rejected due to other reasons","messageMarkedAsOther@board:BLOG":"This post has been rejected due to other reasons","messageMarkedAsOther@board:FORUM":"This discussion has been rejected due to other reasons","messageMarkedAsOther@board:OCCASION":"This event has been rejected due to other reasons","messageMarkedAsOther@board:IDEA":"This idea has been rejected due to other reasons","messageArchived":"This post was archived on {date}","relatedUrl":"View Related Content","relatedContentText":"Showing related content","archivedContentLink":"View Archived Content"},"localOverride":false},"Category:category:Exchange":{"__typename":"Category","id":"category:Exchange","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:Planner":{"__typename":"Category","id":"category:Planner","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:Outlook":{"__typename":"Category","id":"category:Outlook","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:Community-Info-Center":{"__typename":"Category","id":"category:Community-Info-Center","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:EducationSector":{"__typename":"Category","id":"category:EducationSector","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:DrivingAdoption":{"__typename":"Category","id":"category:DrivingAdoption","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:Azure":{"__typename":"Category","id":"category:Azure","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:Windows-Server":{"__typename":"Category","id":"category:Windows-Server","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:SQL-Server":{"__typename":"Category","id":"category:SQL-Server","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:MicrosoftTeams":{"__typename":"Category","id":"category:MicrosoftTeams","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:PublicSector":{"__typename":"Category","id":"category:PublicSector","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:microsoft365":{"__typename":"Category","id":"category:microsoft365","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:IoT":{"__typename":"Category","id":"category:IoT","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:HealthcareAndLifeSciences":{"__typename":"Category","id":"category:HealthcareAndLifeSciences","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:SMB":{"__typename":"Category","id":"category:SMB","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:ITOpsTalk":{"__typename":"Category","id":"category:ITOpsTalk","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:microsoft-endpoint-manager":{"__typename":"Category","id":"category:microsoft-endpoint-manager","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:MicrosoftLearn":{"__typename":"Category","id":"category:MicrosoftLearn","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Blog:board:MicrosoftLearnBlog":{"__typename":"Blog","id":"board:MicrosoftLearnBlog","blogPolicies":{"__typename":"BlogPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}},"boardPolicies":{"__typename":"BoardPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:AI":{"__typename":"Category","id":"category:AI","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:MicrosoftMechanics":{"__typename":"Category","id":"category:MicrosoftMechanics","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:StartupsatMicrosoft":{"__typename":"Category","id":"category:StartupsatMicrosoft","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:PartnerCommunity":{"__typename":"Category","id":"category:PartnerCommunity","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:Windows":{"__typename":"Category","id":"category:Windows","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"Category:category:microsoft-security":{"__typename":"Category","id":"category:microsoft-security","categoryPolicies":{"__typename":"CategoryPolicies","canReadNode":{"__typename":"PolicyResult","failureReason":null}}},"QueryVariables:TopicReplyList:message:3967434:12":{"__typename":"QueryVariables","id":"TopicReplyList:message:3967434:12","value":{"id":"message:3967434","first":10,"sorts":{"postTime":{"direction":"DESC"}},"repliesFirst":3,"repliesFirstDepthThree":1,"repliesSorts":{"postTime":{"direction":"DESC"}},"useAvatar":true,"useAuthorLogin":true,"useAuthorRank":true,"useBody":true,"useKudosCount":true,"useTimeToRead":false,"useMedia":false,"useReadOnlyIcon":false,"useRepliesCount":true,"useSearchSnippet":false,"useAcceptedSolutionButton":false,"useSolvedBadge":false,"useAttachments":false,"attachmentsFirst":5,"useTags":true,"useNodeAncestors":false,"useUserHoverCard":false,"useNodeHoverCard":false,"useModerationStatus":true,"usePreviewSubjectModal":false,"useMessageStatus":true}},"ROOT_MUTATION":{"__typename":"Mutation"},"CachedAsset:text:en_US-components/community/Navbar-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/community/Navbar-1737571274000","value":{"community":"Community Home","inbox":"Inbox","manageContent":"Manage Content","tos":"Terms of Service","forgotPassword":"Forgot Password","themeEditor":"Theme Editor","edit":"Edit Navigation Bar","skipContent":"Skip to content","gxcuf89792":"Tech Community","external-1":"Events","s-m-b":"Small and Medium Businesses","windows-server":"Windows Server","education-sector":"Education Sector","driving-adoption":"Driving Adoption","microsoft-learn":"Microsoft Learn","s-q-l-server":"SQL Server","partner-community":"Microsoft Partner Community","microsoft365":"Microsoft 365","external-9":".NET","external-8":"Teams","external-7":"Github","products-services":"Products","external-6":"Power Platform","communities-1":"Topics","external-5":"Microsoft Security","planner":"Planner","external-4":"Microsoft 365","external-3":"Dynamics 365","azure":"Azure","healthcare-and-life-sciences":"Healthcare and Life Sciences","external-2":"Azure","microsoft-mechanics":"Microsoft Mechanics","microsoft-learn-1":"Community","external-10":"Learning Room Directory","microsoft-learn-blog":"Blog","windows":"Windows","i-t-ops-talk":"ITOps Talk","external-link-1":"View All","microsoft-securityand-compliance":"Microsoft Security","public-sector":"Public Sector","community-info-center":"Lounge","external-link-2":"View All","microsoft-teams":"Microsoft Teams","external":"Blogs","microsoft-endpoint-manager":"Microsoft Intune and Configuration Manager","startupsat-microsoft":"Startups at Microsoft","exchange":"Exchange","a-i":"AI and Machine Learning","io-t":"Internet of Things (IoT)","outlook":"Outlook","external-link":"Community Hubs","communities":"Products"},"localOverride":false},"CachedAsset:text:en_US-components/community/NavbarHamburgerDropdown-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/community/NavbarHamburgerDropdown-1737571274000","value":{"hamburgerLabel":"Side Menu"},"localOverride":false},"CachedAsset:text:en_US-components/community/BrandLogo-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/community/BrandLogo-1737571274000","value":{"logoAlt":"Khoros","themeLogoAlt":"Brand Logo"},"localOverride":false},"CachedAsset:text:en_US-components/community/NavbarTextLinks-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/community/NavbarTextLinks-1737571274000","value":{"more":"More"},"localOverride":false},"CachedAsset:text:en_US-components/authentication/AuthenticationLink-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/authentication/AuthenticationLink-1737571274000","value":{"title.login":"Sign In","title.registration":"Register","title.forgotPassword":"Forgot Password","title.multiAuthLogin":"Sign In"},"localOverride":false},"CachedAsset:text:en_US-components/nodes/NodeLink-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/nodes/NodeLink-1737571274000","value":{"place":"Place {name}"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageView/MessageViewStandard-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageView/MessageViewStandard-1737571274000","value":{"anonymous":"Anonymous","author":"{messageAuthorLogin}","authorBy":"{messageAuthorLogin}","board":"{messageBoardTitle}","replyToUser":" to {parentAuthor}","showMoreReplies":"Show More","replyText":"Reply","repliesText":"Replies","markedAsSolved":"Marked as Solved","movedMessagePlaceholder.BLOG":"{count, plural, =0 {This comment has been} other {These comments have been} }","movedMessagePlaceholder.TKB":"{count, plural, =0 {This comment has been} other {These comments have been} }","movedMessagePlaceholder.FORUM":"{count, plural, =0 {This reply has been} other {These replies have been} }","movedMessagePlaceholder.IDEA":"{count, plural, =0 {This comment has been} other {These comments have been} }","movedMessagePlaceholder.OCCASION":"{count, plural, =0 {This comment has been} other {These comments have been} }","movedMessagePlaceholderUrlText":"moved.","messageStatus":"Status: ","statusChanged":"Status changed: {previousStatus} to {currentStatus}","statusAdded":"Status added: {status}","statusRemoved":"Status removed: {status}","labelExpand":"expand replies","labelCollapse":"collapse replies","unhelpfulReason.reason1":"Content is outdated","unhelpfulReason.reason2":"Article is missing information","unhelpfulReason.reason3":"Content is for a different Product","unhelpfulReason.reason4":"Doesn't match what I was searching for"},"localOverride":false},"CachedAsset:text:en_US-components/messages/ThreadedReplyList-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/messages/ThreadedReplyList-1737571274000","value":{"title":"{count, plural, one{# Reply} other{# Replies}}","title@board:BLOG":"{count, plural, one{# Comment} other{# Comments}}","title@board:TKB":"{count, plural, one{# Comment} other{# Comments}}","title@board:IDEA":"{count, plural, one{# Comment} other{# Comments}}","title@board:OCCASION":"{count, plural, one{# Comment} other{# Comments}}","noRepliesTitle":"No Replies","noRepliesTitle@board:BLOG":"No Comments","noRepliesTitle@board:TKB":"No Comments","noRepliesTitle@board:IDEA":"No Comments","noRepliesTitle@board:OCCASION":"No Comments","noRepliesDescription":"Be the first to reply","noRepliesDescription@board:BLOG":"Be the first to comment","noRepliesDescription@board:TKB":"Be the first to comment","noRepliesDescription@board:IDEA":"Be the first to comment","noRepliesDescription@board:OCCASION":"Be the first to comment","messageReadOnlyAlert:BLOG":"Comments have been turned off for this post","messageReadOnlyAlert:TKB":"Comments have been turned off for this article","messageReadOnlyAlert:IDEA":"Comments have been turned off for this idea","messageReadOnlyAlert:FORUM":"Replies have been turned off for this discussion","messageReadOnlyAlert:OCCASION":"Comments have been turned off for this event"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageReplyCallToAction-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageReplyCallToAction-1737571274000","value":{"leaveReply":"Leave a reply...","leaveReply@board:BLOG@message:root":"Leave a comment...","leaveReply@board:TKB@message:root":"Leave a comment...","leaveReply@board:IDEA@message:root":"Leave a comment...","leaveReply@board:OCCASION@message:root":"Leave a comment...","repliesTurnedOff.FORUM":"Replies are turned off for this topic","repliesTurnedOff.BLOG":"Comments are turned off for this topic","repliesTurnedOff.TKB":"Comments are turned off for this topic","repliesTurnedOff.IDEA":"Comments are turned off for this topic","repliesTurnedOff.OCCASION":"Comments are turned off for this topic","infoText":"Stop poking me!"},"localOverride":false},"Rank:rank:37":{"__typename":"Rank","id":"rank:37","position":18,"name":"Copper Contributor","color":"333333","icon":null,"rankStyle":"TEXT"},"User:user:2122450":{"__typename":"User","id":"user:2122450","uid":2122450,"login":"dewa55","biography":null,"registrationData":{"__typename":"RegistrationData","status":null,"registrationTime":"2023-11-03T11:23:24.786-07:00"},"deleted":false,"email":"","avatar":{"__typename":"UserAvatar","url":"https://techcommunity.microsoft.com/t5/s/gxcuf89792/m_assets/avatars/default/avatar-5.svg?time=0"},"rank":{"__ref":"Rank:rank:37"},"entityType":"USER","eventPath":"community:gxcuf89792/user:2122450"},"ModerationData:moderation_data:3984515":{"__typename":"ModerationData","id":"moderation_data:3984515","status":"APPROVED","rejectReason":null,"isReportedAbuse":false,"rejectUser":null,"rejectTime":null,"rejectActorType":null},"BlogReplyMessage:message:3984515":{"__typename":"BlogReplyMessage","author":{"__ref":"User:user:2122450"},"id":"message:3984515","revisionNum":1,"uid":3984515,"depth":1,"hasGivenKudo":false,"subscribed":false,"board":{"__ref":"Blog:board:FastTrackforAzureBlog"},"parent":{"__ref":"BlogTopicMessage:message:3967434"},"conversation":{"__ref":"Conversation:conversation:3967434"},"subject":"Re: Deploying an Azure Kubernetes Service (AKS) Cluster with Application Gateway for Containers","moderationData":{"__ref":"ModerationData:moderation_data:3984515"},"body":"
","body@stripHtml({\"removeProcessingText\":false,\"removeSpoilerMarkup\":false,\"removeTocMarkup\":false,\"truncateLength\":200})@stringLength":"57","kudosSumWeight":0,"repliesCount":0,"postTime":"2023-11-15T23:30:55.019-08:00","lastPublishTime":"2023-11-15T23:30:55.019-08:00","metrics":{"__typename":"MessageMetrics","views":5143},"visibilityScope":"PUBLIC","placeholder":false,"originalMessageForPlaceholder":null,"entityType":"BLOG_REPLY","eventPath":"category:FastTrack/category:products-services/category:communities/community:gxcuf89792board:FastTrackforAzureBlog/message:3967434/message:3984515","replies":{"__typename":"MessageConnection","pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null},"edges":[]},"customFields":[],"attachments":{"__typename":"AttachmentConnection","edges":[],"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}}},"Rank:rank:35":{"__typename":"Rank","id":"rank:35","position":16,"name":"Iron Contributor","color":"333333","icon":null,"rankStyle":"TEXT"},"User:user:50793":{"__typename":"User","id":"user:50793","uid":50793,"login":"Darifer","biography":null,"registrationData":{"__typename":"RegistrationData","status":null,"registrationTime":"2017-04-13T01:46:42.383-07:00"},"deleted":false,"email":"","avatar":{"__typename":"UserAvatar","url":"https://techcommunity.microsoft.com/t5/s/gxcuf89792/images/dS01MDc5My01OTA2MjVpMTZGRDRFNUREODYwNkUxOA"},"rank":{"__ref":"Rank:rank:35"},"entityType":"USER","eventPath":"community:gxcuf89792/user:50793"},"ModerationData:moderation_data:3968544":{"__typename":"ModerationData","id":"moderation_data:3968544","status":"APPROVED","rejectReason":null,"isReportedAbuse":false,"rejectUser":null,"rejectTime":null,"rejectActorType":null},"BlogReplyMessage:message:3968544":{"__typename":"BlogReplyMessage","author":{"__ref":"User:user:50793"},"id":"message:3968544","revisionNum":1,"uid":3968544,"depth":1,"hasGivenKudo":false,"subscribed":false,"board":{"__ref":"Blog:board:FastTrackforAzureBlog"},"parent":{"__ref":"BlogTopicMessage:message:3967434"},"conversation":{"__ref":"Conversation:conversation:3967434"},"subject":"Re: Deploying an Azure Kubernetes Service (AKS) Cluster with Application Gateway for Containers","moderationData":{"__ref":"ModerationData:moderation_data:3968544"},"body":"
","body@stripHtml({\"removeProcessingText\":false,\"removeSpoilerMarkup\":false,\"removeTocMarkup\":false,\"truncateLength\":200})@stringLength":"60","kudosSumWeight":1,"repliesCount":0,"postTime":"2023-10-31T03:46:26.288-07:00","lastPublishTime":"2023-10-31T03:46:26.288-07:00","metrics":{"__typename":"MessageMetrics","views":7565},"visibilityScope":"PUBLIC","placeholder":false,"originalMessageForPlaceholder":null,"entityType":"BLOG_REPLY","eventPath":"category:FastTrack/category:products-services/category:communities/community:gxcuf89792board:FastTrackforAzureBlog/message:3967434/message:3968544","replies":{"__typename":"MessageConnection","pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null},"edges":[]},"customFields":[],"attachments":{"__typename":"AttachmentConnection","edges":[],"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}}},"ModerationData:moderation_data:3967479":{"__typename":"ModerationData","id":"moderation_data:3967479","status":"APPROVED","rejectReason":null,"isReportedAbuse":false,"rejectUser":null,"rejectTime":null,"rejectActorType":null},"BlogReplyMessage:message:3967479":{"__typename":"BlogReplyMessage","author":{"__ref":"User:user:988334"},"id":"message:3967479","revisionNum":1,"uid":3967479,"depth":1,"hasGivenKudo":false,"subscribed":false,"board":{"__ref":"Blog:board:FastTrackforAzureBlog"},"parent":{"__ref":"BlogTopicMessage:message:3967434"},"conversation":{"__ref":"Conversation:conversation:3967434"},"subject":"Re: Deploying an Azure Kubernetes Service (AKS) Cluster with Application Gateway for Containers","moderationData":{"__ref":"ModerationData:moderation_data:3967479"},"body":"
Thanks JamesvandenBerg, you know that you can always count on my contribution!
","body@stripHtml({\"removeProcessingText\":false,\"removeSpoilerMarkup\":false,\"removeTocMarkup\":false,\"truncateLength\":200})@stringLength":"90","kudosSumWeight":1,"repliesCount":0,"postTime":"2023-10-30T02:33:33.976-07:00","lastPublishTime":"2023-10-30T02:33:33.976-07:00","metrics":{"__typename":"MessageMetrics","views":9051},"visibilityScope":"PUBLIC","placeholder":false,"originalMessageForPlaceholder":null,"entityType":"BLOG_REPLY","eventPath":"category:FastTrack/category:products-services/category:communities/community:gxcuf89792board:FastTrackforAzureBlog/message:3967434/message:3967479","replies":{"__typename":"MessageConnection","pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null},"edges":[]},"customFields":[],"attachments":{"__typename":"AttachmentConnection","edges":[],"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}}},"Rank:rank:5":{"__typename":"Rank","id":"rank:5","position":7,"name":"MVP","color":"0069D4","icon":null,"rankStyle":"FILLED"},"User:user:9011":{"__typename":"User","id":"user:9011","uid":9011,"login":"JamesvandenBerg","biography":null,"registrationData":{"__typename":"RegistrationData","status":null,"registrationTime":"2016-09-02T11:14:33.449-07:00"},"deleted":false,"email":"","avatar":{"__typename":"UserAvatar","url":"https://techcommunity.microsoft.com/t5/s/gxcuf89792/images/dS05MDExLTI4MDdpRUNDN0I3N0RBQzJDMTMyRg"},"rank":{"__ref":"Rank:rank:5"},"entityType":"USER","eventPath":"community:gxcuf89792/user:9011"},"ModerationData:moderation_data:3967457":{"__typename":"ModerationData","id":"moderation_data:3967457","status":"APPROVED","rejectReason":null,"isReportedAbuse":false,"rejectUser":null,"rejectTime":null,"rejectActorType":null},"BlogReplyMessage:message:3967457":{"__typename":"BlogReplyMessage","author":{"__ref":"User:user:9011"},"id":"message:3967457","revisionNum":1,"uid":3967457,"depth":1,"hasGivenKudo":false,"subscribed":false,"board":{"__ref":"Blog:board:FastTrackforAzureBlog"},"parent":{"__ref":"BlogTopicMessage:message:3967434"},"conversation":{"__ref":"Conversation:conversation:3967434"},"subject":"Re: Deploying an Azure Kubernetes Service (AKS) Cluster with Application Gateway for Containers","moderationData":{"__ref":"ModerationData:moderation_data:3967457"},"body":"
Thank you paolosalvatori for Sharing this Awesome Blogpost with the Tech Community
","body@stripHtml({\"removeProcessingText\":false,\"removeSpoilerMarkup\":false,\"removeTocMarkup\":false,\"truncateLength\":200})@stringLength":"102","kudosSumWeight":1,"repliesCount":0,"postTime":"2023-10-30T02:11:22.800-07:00","lastPublishTime":"2023-10-30T02:11:22.800-07:00","metrics":{"__typename":"MessageMetrics","views":9085},"visibilityScope":"PUBLIC","placeholder":false,"originalMessageForPlaceholder":null,"entityType":"BLOG_REPLY","eventPath":"category:FastTrack/category:products-services/category:communities/community:gxcuf89792board:FastTrackforAzureBlog/message:3967434/message:3967457","replies":{"__typename":"MessageConnection","pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null},"edges":[]},"customFields":[],"attachments":{"__typename":"AttachmentConnection","edges":[],"pageInfo":{"__typename":"PageInfo","hasNextPage":false,"endCursor":null,"hasPreviousPage":false,"startCursor":null}}},"CachedAsset:text:en_US-components/community/NavbarDropdownToggle-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/community/NavbarDropdownToggle-1737571274000","value":{"ariaLabelClosed":"Press the down arrow to open the menu"},"localOverride":false},"CachedAsset:text:en_US-shared/client/components/common/QueryHandler-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-shared/client/components/common/QueryHandler-1737571274000","value":{"title":"Query Handler"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageCoverImage-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageCoverImage-1737571274000","value":{"coverImageTitle":"Cover Image"},"localOverride":false},"CachedAsset:text:en_US-shared/client/components/nodes/NodeTitle-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-shared/client/components/nodes/NodeTitle-1737571274000","value":{"nodeTitle":"{nodeTitle, select, community {Community} other {{nodeTitle}}} "},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageTimeToRead-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageTimeToRead-1737571274000","value":{"minReadText":"{min} MIN READ"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageSubject-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageSubject-1737571274000","value":{"noSubject":"(no subject)"},"localOverride":false},"CachedAsset:text:en_US-components/users/UserLink-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/users/UserLink-1737571274000","value":{"authorName":"View Profile: {author}","anonymous":"Anonymous"},"localOverride":false},"CachedAsset:text:en_US-shared/client/components/users/UserRank-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-shared/client/components/users/UserRank-1737571274000","value":{"rankName":"{rankName}","userRank":"Author rank {rankName}"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageTime-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageTime-1737571274000","value":{"postTime":"Published: {time}","lastPublishTime":"Last Update: {time}","conversation.lastPostingActivityTime":"Last posting activity time: {time}","conversation.lastPostTime":"Last post time: {time}","moderationData.rejectTime":"Rejected time: {time}"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageBody-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageBody-1737571274000","value":{"showMessageBody":"Show More","mentionsErrorTitle":"{mentionsType, select, board {Board} user {User} message {Message} other {}} No Longer Available","mentionsErrorMessage":"The {mentionsType} you are trying to view has been removed from the community.","videoProcessing":"Video is being processed. Please try again in a few minutes.","bannerTitle":"Video provider requires cookies to play the video. Accept to continue or {url} it directly on the provider's site.","buttonTitle":"Accept","urlText":"watch"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageCustomFields-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageCustomFields-1737571274000","value":{"CustomField.default.label":"Value of {name}"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageRevision-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageRevision-1737571274000","value":{"lastUpdatedDatePublished":"{publishCount, plural, one{Published} other{Updated}} {date}","lastUpdatedDateDraft":"Created {date}","version":"Version {major}.{minor}"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageReplyButton-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageReplyButton-1737571274000","value":{"repliesCount":"{count}","title":"Reply","title@board:BLOG@message:root":"Comment","title@board:TKB@message:root":"Comment","title@board:IDEA@message:root":"Comment","title@board:OCCASION@message:root":"Comment"},"localOverride":false},"CachedAsset:text:en_US-components/messages/MessageAuthorBio-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/messages/MessageAuthorBio-1737571274000","value":{"sendMessage":"Send Message","actionMessage":"Follow this blog board to get notified when there's new activity","coAuthor":"CO-PUBLISHER","contributor":"CONTRIBUTOR","userProfile":"View Profile","iconlink":"Go to {name} {type}"},"localOverride":false},"CachedAsset:text:en_US-shared/client/components/users/UserAvatar-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-shared/client/components/users/UserAvatar-1737571274000","value":{"altText":"{login}'s avatar","altTextGeneric":"User's avatar"},"localOverride":false},"CachedAsset:text:en_US-shared/client/components/ranks/UserRankLabel-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-shared/client/components/ranks/UserRankLabel-1737571274000","value":{"altTitle":"Icon for {rankName} rank"},"localOverride":false},"CachedAsset:text:en_US-components/users/UserRegistrationDate-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/users/UserRegistrationDate-1737571274000","value":{"noPrefix":"{date}","withPrefix":"Joined {date}"},"localOverride":false},"CachedAsset:text:en_US-shared/client/components/nodes/NodeAvatar-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-shared/client/components/nodes/NodeAvatar-1737571274000","value":{"altTitle":"Node avatar for {nodeTitle}"},"localOverride":false},"CachedAsset:text:en_US-shared/client/components/nodes/NodeDescription-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-shared/client/components/nodes/NodeDescription-1737571274000","value":{"description":"{description}"},"localOverride":false},"CachedAsset:text:en_US-components/tags/TagView/TagViewChip-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-components/tags/TagView/TagViewChip-1737571274000","value":{"tagLabelName":"Tag name {tagName}"},"localOverride":false},"CachedAsset:text:en_US-shared/client/components/nodes/NodeIcon-1737571274000":{"__typename":"CachedAsset","id":"text:en_US-shared/client/components/nodes/NodeIcon-1737571274000","value":{"contentType":"Content Type {style, select, FORUM {Forum} BLOG {Blog} TKB {Knowledge Base} IDEA {Ideas} OCCASION {Events} other {}} icon"},"localOverride":false}}}},"page":"/blogs/BlogMessagePage/BlogMessagePage","query":{"boardId":"fasttrackforazureblog","messageSubject":"deploying-an-azure-kubernetes-service-aks-cluster-with-application-gateway-for-c","messageId":"3967434"},"buildId":"HEhyUrv5OXNBIbfCLaOrw","runtimeConfig":{"buildInformationVisible":false,"logLevelApp":"info","logLevelMetrics":"info","openTelemetryClientEnabled":false,"openTelemetryConfigName":"o365","openTelemetryServiceVersion":"25.1.0","openTelemetryUniverse":"prod","openTelemetryCollector":"http://localhost:4318","openTelemetryRouteChangeAllowedTime":"5000","apolloDevToolsEnabled":false,"inboxMuteWipFeatureEnabled":false},"isFallback":false,"isExperimentalCompile":false,"dynamicIds":["./components/community/Navbar/NavbarWidget.tsx","./components/community/Breadcrumb/BreadcrumbWidget.tsx","./components/customComponent/CustomComponent/CustomComponent.tsx","./components/blogs/BlogArticleWidget/BlogArticleWidget.tsx","./components/external/components/ExternalComponent.tsx","./components/messages/MessageView/MessageViewStandard/MessageViewStandard.tsx","./components/messages/ThreadedReplyList/ThreadedReplyList.tsx","../shared/client/components/common/List/UnstyledList/UnstyledList.tsx","./components/messages/MessageView/MessageView.tsx","../shared/client/components/common/List/UnwrappedList/UnwrappedList.tsx","./components/tags/TagView/TagView.tsx","./components/tags/TagView/TagViewChip/TagViewChip.tsx"],"appGip":true,"scriptLoader":[{"id":"analytics","src":"https://techcommunity.microsoft.com/t5/s/gxcuf89792/pagescripts/1730819800000/analytics.js?page.id=BlogMessagePage&entity.id=board%3Afasttrackforazureblog&entity.id=message%3A3967434","strategy":"afterInteractive"}]}
Microsoft
![\":stareyes:\"](\"http://techcommunity.microsoft.com/t5/s/html/@7C3DD6040F950CF276CFDB3C1D92ABBC/images/emoticons/stareyes_40x40.gif\" "\\":stareyes:\\"")