gMSA on AKS and Private Endpoints
A few weeks ago, I spent some time with our support and engineering teams helping a customer solve a problem that happened after they enabled Group Managed Service Accounts (gMSA) on Azure Kubernetes Service (AKS). I decided to write this blog so other customers with the same issue can avoid going through it altogether. I’m writing the blog in the sequence as I experienced it, but if you’re just looking for the solution, feel free to skip to the end. The gMSA on AKS symptoms When that customer enabled gMSA on their cluster, a few things started to happen: Any gMSA enabled deployment/container/pod entered a failed state. The events from the deployments would show the pods with the following error: Event Detail: Failed to setup the external credentials for Container '<redacted>': The RPC server is unavailable. Any non-gMSA deployment/container/pod using the customer’s private images and running on Windows nodes also entered a failed state. The deployments were showing an event of ErrImagePull. All other deployments/containers/pods both on Windows and Linux nodes that were not using private images kept their healthy state. Removing the gMSA configuration from the cluster would automatically revert to a healthy state for the entire cluster. Troubleshooting gMSA issues The error with the gMSA pods took me immediately to other cases in which I’ve seen customers having similar issues because of network connectivity. The most common gMSA issues I have seen so far are: Blocked ports: Having a firewall between your AKS cluster and the Active Directory (AD) Domain Controllers (DCs). AD uses multiple protocols for communication between clients and DCs. I even created a simple script that validates the ports. Incorrect DNS configuration: AD uses DNS for service discovery. Domain Controllers have a “SRV” entry in the DNS that clients query so they can find not only all DCs, but the closest one. If either the nodes or pods can’t resolve the domain fqdn to a DC, gMSA won’t work. Incorrect secret on Azure Key Vault (AKV): A user account is used by the Window nodes, rather than a computer account as the nodes are not domain-joined. The format of the secret should be <domain dns fqdn>\<user account>:<user password>. There are other minor issues that I’ve seen, but these are the main ones. In the case of this customers, we reviewed the above and everything seemed to be configured properly. At that point, I brought other folks and they caught on something that I knew existed, but had not seen using gMSA yet: AKS private clusters. Private Endpoints and gMSA This customer has a security policy in-place that mandates Azure resources should be using private endpoints whenever possible. That was true for the AKS cluster and therefore, it introduced a behavior that broke the cluster. I mentioned above that gMSA uses DNS for DC finding. Let me explain what the default config is and what happened after enabling gMSA: By default, Linux and Windows nodes on AKS will use the Azure vNet DNS server for DNS queries. Windows and Linux pods will use CoreDNS for DNS queries. Azure DNS can’t resolve AD domain FQDNs since these tend to be private to on-premises or non-public cloud networks. For that reason, when you enable gMSA and pass the parameter of DNS server to be used, two things are changed in the AKS cluster. First, the Windows nodes will start using the DNS server provided. Second, the CoreDNS setting is changed to add a forwarder. This forwards anything related to the domain FQDN to the specified DNS server. With these two configs, Windows nodes and Windows pods can now “find” the DCs. However, this introduces another issue when combined with a private AKS cluster. Private endpoints are behind a private DNS zone. Azure DNS servers can resolve for those zones, but non-Azure DNS servers can’t. Since now the Windows nodes and Windows pods are using a DNS server outside of Azure, the private zone of the AKS cluster can’t be resolved so the DCs can’t access the Windows nodes and Windows pods. Not only that, but this customer also had their Azure Container Registry (ACR) behind a private endpoint. The second symptom above was also caused by this configuration, as now the Windows nodes can’t resolve for the private zone of the ACR registry and consequently can’t pull their private images. For reference, these are the container related services and their private zones: Private link resource type Subresource Private DNS zone name Public DNS zone forwarders Azure Kubernetes Service - Kubernetes API (Microsoft.ContainerService/managedClusters) management privatelink.{regionName}.azmk8s.io {subzone}.privatelink.{regionName}.azmk8s.io {regionName}.azmk8s.io Azure Container Apps (Microsoft.App/ManagedEnvironments) managedEnvironments privatelink.{regionName}.azurecontainerapps.io azurecontainerapps.io Azure Container Registry (Microsoft.ContainerRegistry/registries) registry privatelink.azurecr.io {regionName}.data.privatelink.azurecr.io azurecr.io {regionName}.data.azurecr.io For a full list of zones, check out the Azure documentation. Solving for DNS query on Azure Private Endpoint zones The solution here is simple. For the non-Azure DNS servers to resolve Private Endpoint zones, a DNS forwarder can be created. This customer had a very specific implementation, but in general what you need to configure is a DNS forwarder to the zones related to the services you are using. For example: - AKS clusters: Create a forwarder of azmk8s.io to 168.63.129.16. - For ACR registries: Create a forwarder of azurecr.io to 168.63.129.16. 168.63.129.16. is the virtual IP address of the Azure platform that serves as the communication channel to the platform resources. One of its services is DNS. In fact, this is the original service that the Windows nodes and Windows pods were using before gMSA was enabled. Conclusion It’s always DNS! If you are using gMSA on AKS, keep in mind that Windows nodes and Windows pods will start using a DNS server outside of Azure (or that has no visibility into the Azure platform directly, such as Private Endpoint zones). You might need to configure DNS forwarders once you start using gMSA on AKS, although this will be true for any service. I hope this blog post helps you avoid this issue – or helps you troubleshoot it. Let us know in the comments!Step-by-Step Guide : How to enable QR code authentication for Microsoft Entra ID (Preview) ?
Microsoft Entra ID supports a long list of Authentication methods. Windows Hello for Business Microsoft Authenticator app Authenticator Lite Passkey (FIDO2) Certificate-based authentication Hardware OATH tokens (preview) Software OATH tokens External authentication methods (preview) Temporary Access Pass (TAP) Short Message Service (SMS) sign-in and verification Voice call verification Password This enables organizations to select the most secure and productive authentication methods for their business. While the most secure method may not always be the most productive, and vice versa, having a variety of supported authentication methods helps to strike a balance between these two aspects. Microsoft Entra ID now supports QR authentication, a method specifically designed for frontline workers who use shared devices. This provides a convenient and secure login experience for these workers. How it works ? 1) An account with Authentication Policy Administrator permission or higher can enable QR code as an authentication method. 2) Once the method is enabled, a QR code and temporary PIN can be generated for the user. 3) The QR code should be made available to the user. It can be downloaded, printed, or added to a badge. 4) The QR code is unique but cannot be used without the PIN. 5) The temporary PIN must be reset when the user authenticates for the first time. 6) Once the QR code and PIN are set up, the user can use them for subsequent logins. Things to remember ! 1) QR authentication is designed for frontline workers and should not be widely used. Phishing-resistant authentication is recommended wherever possible. 2) Do not enable this authentication method for all users; only enable it for required users. 3) QR authentication is currently only supported on mobile devices running iOS/iPadOS or Android. 4) QR authentication does not allow self-service PIN reset for users. In this blog post I am going to demonstrate how to configure QR authentication for the Microsoft Entra ID users. Let’s start with enabling authentication method. Log in to the Entra admin portal at https://entra.microsoft.com/as an Authentication Policy Administrator or higher. Navigate to Protection | Authentication Methods. Under Policies, click on QR code (Preview). In the QR code (Preview) settings page, click on Enable to turn on the authentication method. Then, select the relevant user group as the target. Click on the Configure tab. Here, you can adjust the PIN length and the lifetime of the QR code. The default is 365 days, but it can be extended up to 395 days. Once changes are made, click on Save to apply them. This enables the QR code as an authentication method for the tenant. Next, let's see how to generate a QR code for a user. Generate QR code authentication for a user To generate QR code for user, Navigate to Users | All users. Select the user from the target group configured in the previous section. Click on Authentication methods. Click on + Add authentication method. From the dropdown, select QR code (Preview). In the settings page, define the expiration date and activation time. Click on Generate PIN to create a temporary PIN. Note down the PIN and click on Add. This will generate the QR code. Download it for use with authentication. Now that we have generated a QR code for a user, let's proceed with some testing. Testing For testing, I used an iOS device to log in to the office portal. On the login page, I typed the username and then clicked on Sign-in options. In the Sign-in options page, I selected Sign in to an organization. On the next page, I chose Sign in with QR code. I clicked on Allow to grant access to the camera. After that, I scanned the QR code downloaded in the previous step. Once the QR code was successfully detected, I entered the temporary PIN that was generated and clicked on Sign in. On the next page, I was prompted to define a new PIN since this was the first login. After defining the PIN, I clicked on Sign in. As expected, I was able to log in successfully. This marks the end of the blog post, and I believe you now have a better understanding of how to enable and use QR code for authentication.Step-by-Step Guide : How to use Temporary Access Pass (TAP) with internal guest users
Passwords are fundamentally weak and vulnerable to being compromised. Even enhancing a password only delays an attack; it does not render it unbreakable. Multi-Factor Authentication (MFA) offers more security but still depends on passwords. This is why passwordless authentication is a more secure and convenient alternative. Source : https://learn.microsoft.com/entra/identity/authentication/media/concept-authentication-passwordless/passwordless-convenience-security.png Microsoft Entra ID supports password less authentication natively. It supports six different password less authentication options. Windows Hello for Business Platform Credential for macOS Platform single sign-on (PSSO) for macOS with smart card authentication Microsoft Authenticator Passkeys (FIDO2) Certificate-based authentication Based on the organisation's requirements, they can select the most convenient options. However, the initial setup requires a method to authenticate the user before onboarding other passwordless authentication methods. For this, we can use: 1) Existing Microsoft MFA methods 2) Temporary Access pass (TAP) A Temporary Access Pass (TAP) is a time-limited passcode that can be configured for single use or multiple sign-ins. Organisations not only have internal users to manage but also guest users. Until now, the TAP method was only available for internal users, and guest users were not permitted to use this method. This makes sense because if guest users also need to use passwordless authentication, it should occur in their home tenant. But now Entra ID supports TAP for “Internal Guest” users. Internal Guests Guest users are typically categorised as user accounts that exist in a remote tenant. However, some organisations prefer to use user accounts in their own directory but with guest-level access. This is typically for contractors, suppliers, vendors, etc. These are known as 'internal guest accounts'. Such accounts were also used for guest users in the past when B2B collaboration wasn't in place. In this demo I am going to demonstrate how to use TAP with internal Guest user. Enable TAP as Authentication method Before we configure TAP for user we need to make sure TAP is enabled as authentication method. To do that, Log in to the Entra portal as an Authentication Policy Administrator or higher. Navigate to Protection > Authentication methods > Policies. Click on Temporary Access Pass Ensure it is enabled and the target is defined. If not, make the necessary changes and click Save. Create TAP for Internal Guest User I already have an internal guest user for this task. As you can see below, the user type is Guest, but the user is still part of the same tenant. To create TAP, Click on the selected user from the Entra ID users list to go to user properties. Next, Click on Authentication methods Then Click on + Add authentication method From the drop-down, select the Temporary Access Pass method. In the settings window, make the adjustments based on the requirements and then click on Add. It will create TAP as expected. Testing To verify the configuration, I am attempting to log in as the test user. This is the user's very first login. As expected, the initial login prompts for the TAP. After a successful login, it allows me to configure the account with passwordless authentication. As we can see, the TAP for the internal guest feature is working as expected.How to In-Place Upgrade Windows Server 2008 R2 to Windows Server 2019
As you know the Windows Server 2008 and Windows Server 2008 R2 are out of support on January 14th, 2020. Customer will need to upgrade their Windows Server 2008 and Windows Server 2008 R2 to a newer version of Windows Server or migrate these servers to Microsoft Azure.
