While trying out the capabilities of Azure Arc for Servers, I came across a couple of times when the on-premises virtual machine showed as ‘Offline’ in the Azure Portal. To help troubleshoot this, there’s a command line tool that can help reconnect the machine to Azure, called azcmagent.exe.
1. Check the machine state
My experience as a systems administrator have ingrained in me the approach to check the simple things first. In this case, is the virtual machine actually running?
If you have a way to verify connecting to or getting a response from the server another way, do that first. Many things can cause a virtual machine to stop responding, including issues with the host. So we want to first check that the server is running and otherwise healthy.
2. Check the Azure Arc for Servers state
Azure Arc for Servers uses an agent known as the Connected Machine Agent. If installed successfully, you can find it listed in Add or remove programs:
It runs two services: the Azure Hybrid Instance Metadata Service and the Guest Configuration Service.
From a command prompt, we can use the azcmagent command to check the status of the agent:
azcmagent show
With the error below, the root cause was that one of the services was not running.
So, maybe the program was accidentally uninstalled, is being blocked by aggressive third party security software, or maybe the server has other underlying issues preventing the service from running.
Not pictured, it’s also possible to get an Agent Status: Disconnected response. This indicates that things are functioning from an agent perspective, but it’s not communicating with Azure. Our challenge is to re-establish that connection or figure out what’s preventing it.
3. Double check the prerequisites
Azure Arc for Servers has some pre-requisites, including supported operating systems, supported Azure regions and Azure subscription and service limits. If you’ve previously onboarded and connected a machine successfully, make sure none of those requirements listed here, have changed.
There are no special network ports required, as Azure Arc for Servers uses TCP Port 443. If your firewall is very restrictive, you’ll want to check you’re still allowed access to the required Azure URLs (preferably) or IP ranges and service tags. For details, see Networking Configuration.
4. Issue a reconnect command
If your server was previously connected and working, use the azcmagent tool to issue a reconnect command. This is useful if the server has been turned off (for more than 45 days) and its local certificate has expired, but the machine still appears in Azure. It’s also a useful troubleshooting step if the agent status in the Azure Portal is reporting as Offline. This step will require you to provide valid Azure credentials, in the form of a service principal, an access token or via the Azure login screen.
azcmagent reconnect --tenant-id <tenantID>
This interactive login method prompts you to visit https://aka.ms/devicelogin in a browser, log in with your credentials and enter the unique code displayed in the command response. Note: by default, Internet Explorer runs in Enhanced Protected Mode on a Windows Server, which blocks the functionality of this website. To enable it, use Server Manager to bring up the Local Server properties and click IE Enhanced Security Configuration: On, to turn that protection off. Make sure you re-enable it again once you have successfully logged in.
The preferred method is to use a service principal, which requires you to provide:
azcmagent reconnect --service-principal-id <serviceprincipalAppID> --service-principal-secret <serviceprincipalPassword> --tenant-id <tenantID>
If you didn’t generate a service principal when you first registered the machine, you can still do so by following these steps – Create a Service Principal for onboarding at scale.
Then hopefully, our azcmagent show command should get a Connected response, similar to this:
5. Disconnect and Connect
Another use of the azcmagent tool is to remove a server if the Azure Arc Connected Machine resource has been deleted from Azure (especially accidentally), and then to connect it again, creating a new resource. This does not remove or reinstall the agent and it’s services on the virtual machine. Both parameters do require you to provide valid credentials, in the way the disconnect method does.
azcmagent disconnect --service-principal-id <serviceprincipalAppID> --service-principal-secret <serviceprincipalPassword> --tenant-id <tenantID>azcmagent connect --service-principal-id <serviceprincipalAppID> --service-principal-secret <serviceprincipalPassword> --tenant-id <tenantID> --subscription-id <subscriptionID> --resource-group <ResourceGroupName> --location <resourceLocation>
Learn more:
For more information, visit Managing and maintaining the Connected Machine agent.
-SCuffy