How to add a Server to Azure Arc
In this blog post, we are going to have a quick look at Azure Arc for servers and how you can add servers to Azure Arc. Azure Arc for servers extends Azure Resource Manager to allow you to manage machines that are outside of Azure, using Azure management tools like Azure Log Analytics and Azure Policy.How to In-Place Upgrade Windows Server 2008 R2 to Windows Server 2019
As you know the Windows Server 2008 and Windows Server 2008 R2 are out of support on January 14th, 2020. Customer will need to upgrade their Windows Server 2008 and Windows Server 2008 R2 to a newer version of Windows Server or migrate these servers to Microsoft Azure.Announcing the new Microsoft Learn Plan - Preparing for your organization's AI workloads
We're pleased to announce the new "Preparing for your organization's AI workloads" Microsoft Learn Plan - focused the IT/Ops audience and now available on Microsoft Learn! This set of content was curated by our team and is targeted at helping IT Professionals who want to learn how to support their organization's AI applications and infrastructure. The Learning Plan is composed of 4 milestones, which in its turn are composed of a total of 22 modules: Milestone 1: Getting Started with AI on Microsoft Azure - Learning Path: Introduction to AI in Azure - 12 modules Milestone 2: Introduction to AI Services Infrastructure on Azure - Learning Path: Manage Authentication, Authorization, and RBAC for AI Workloads on Azure - 3 modules - Learning Path: Manage Network Access for AI Workloads - 2 modules Milestone 3: Monitoring AI Services on Azure - Learning Path: Monitor AI Workloads on Azure - 3 modules Milestone 4: Advanced Management of AI Workloads on Azure - Learning Path: AI Workload Governance and DLP - 2 modules This comprehensive plan introduces foundational AI concepts, then guides you through advanced topics. Whether you're an IT administrator, security specialist, or AI practitioner, this plan equips you with the skills to build trusted, secure, and compliant AI solutions at scale. We hope you enjoy learning! Let us know what you think about this content in the comment section below! If you'd like to see more of this type of content, or have any suggestions, let us know as well!How to Create Azure Monitor Alerts for Non-Compliant Azure Policies
Azure Policy allows organization to define and enforce how Azure resources should or should not be used. It allows for the ability to block resources with certain configurations from being created, or generate audit events when a particular configuration is used. It can also modify resources at creation stage to ensure it meets standards set by your organization. This blog post detail how to use Kusto query language to detect non-compliant resources by querying your Log Analytics workspace, and then how to create Azure monitor alert rules to take actions when non-compliant resources are detected by Azure Policy.gMSA on AKS and Private Endpoints
A few weeks ago, I spent some time with our support and engineering teams helping a customer solve a problem that happened after they enabled Group Managed Service Accounts (gMSA) on Azure Kubernetes Service (AKS). I decided to write this blog so other customers with the same issue can avoid going through it altogether. I’m writing the blog in the sequence as I experienced it, but if you’re just looking for the solution, feel free to skip to the end. The gMSA on AKS symptoms When that customer enabled gMSA on their cluster, a few things started to happen: Any gMSA enabled deployment/container/pod entered a failed state. The events from the deployments would show the pods with the following error: Event Detail: Failed to setup the external credentials for Container '<redacted>': The RPC server is unavailable. Any non-gMSA deployment/container/pod using the customer’s private images and running on Windows nodes also entered a failed state. The deployments were showing an event of ErrImagePull. All other deployments/containers/pods both on Windows and Linux nodes that were not using private images kept their healthy state. Removing the gMSA configuration from the cluster would automatically revert to a healthy state for the entire cluster. Troubleshooting gMSA issues The error with the gMSA pods took me immediately to other cases in which I’ve seen customers having similar issues because of network connectivity. The most common gMSA issues I have seen so far are: Blocked ports: Having a firewall between your AKS cluster and the Active Directory (AD) Domain Controllers (DCs). AD uses multiple protocols for communication between clients and DCs. I even created a simple script that validates the ports. Incorrect DNS configuration: AD uses DNS for service discovery. Domain Controllers have a “SRV” entry in the DNS that clients query so they can find not only all DCs, but the closest one. If either the nodes or pods can’t resolve the domain fqdn to a DC, gMSA won’t work. Incorrect secret on Azure Key Vault (AKV): A user account is used by the Window nodes, rather than a computer account as the nodes are not domain-joined. The format of the secret should be <domain dns fqdn>\<user account>:<user password>. There are other minor issues that I’ve seen, but these are the main ones. In the case of this customers, we reviewed the above and everything seemed to be configured properly. At that point, I brought other folks and they caught on something that I knew existed, but had not seen using gMSA yet: AKS private clusters. Private Endpoints and gMSA This customer has a security policy in-place that mandates Azure resources should be using private endpoints whenever possible. That was true for the AKS cluster and therefore, it introduced a behavior that broke the cluster. I mentioned above that gMSA uses DNS for DC finding. Let me explain what the default config is and what happened after enabling gMSA: By default, Linux and Windows nodes on AKS will use the Azure vNet DNS server for DNS queries. Windows and Linux pods will use CoreDNS for DNS queries. Azure DNS can’t resolve AD domain FQDNs since these tend to be private to on-premises or non-public cloud networks. For that reason, when you enable gMSA and pass the parameter of DNS server to be used, two things are changed in the AKS cluster. First, the Windows nodes will start using the DNS server provided. Second, the CoreDNS setting is changed to add a forwarder. This forwards anything related to the domain FQDN to the specified DNS server. With these two configs, Windows nodes and Windows pods can now “find” the DCs. However, this introduces another issue when combined with a private AKS cluster. Private endpoints are behind a private DNS zone. Azure DNS servers can resolve for those zones, but non-Azure DNS servers can’t. Since now the Windows nodes and Windows pods are using a DNS server outside of Azure, the private zone of the AKS cluster can’t be resolved so the DCs can’t access the Windows nodes and Windows pods. Not only that, but this customer also had their Azure Container Registry (ACR) behind a private endpoint. The second symptom above was also caused by this configuration, as now the Windows nodes can’t resolve for the private zone of the ACR registry and consequently can’t pull their private images. For reference, these are the container related services and their private zones: Private link resource type Subresource Private DNS zone name Public DNS zone forwarders Azure Kubernetes Service - Kubernetes API (Microsoft.ContainerService/managedClusters) management privatelink.{regionName}.azmk8s.io {subzone}.privatelink.{regionName}.azmk8s.io {regionName}.azmk8s.io Azure Container Apps (Microsoft.App/ManagedEnvironments) managedEnvironments privatelink.{regionName}.azurecontainerapps.io azurecontainerapps.io Azure Container Registry (Microsoft.ContainerRegistry/registries) registry privatelink.azurecr.io {regionName}.data.privatelink.azurecr.io azurecr.io {regionName}.data.azurecr.io For a full list of zones, check out the Azure documentation. Solving for DNS query on Azure Private Endpoint zones The solution here is simple. For the non-Azure DNS servers to resolve Private Endpoint zones, a DNS forwarder can be created. This customer had a very specific implementation, but in general what you need to configure is a DNS forwarder to the zones related to the services you are using. For example: - AKS clusters: Create a forwarder of azmk8s.io to 168.63.129.16. - For ACR registries: Create a forwarder of azurecr.io to 168.63.129.16. 168.63.129.16. is the virtual IP address of the Azure platform that serves as the communication channel to the platform resources. One of its services is DNS. In fact, this is the original service that the Windows nodes and Windows pods were using before gMSA was enabled. Conclusion It’s always DNS! If you are using gMSA on AKS, keep in mind that Windows nodes and Windows pods will start using a DNS server outside of Azure (or that has no visibility into the Azure platform directly, such as Private Endpoint zones). You might need to configure DNS forwarders once you start using gMSA on AKS, although this will be true for any service. I hope this blog post helps you avoid this issue – or helps you troubleshoot it. Let us know in the comments!
