Forum Discussion
Cannot run "Get-SolutionUpdate" or "Get-SolutionUpdateEnvironment" after updating to 2604
My Azure Local environment had been a bit out of date for the last few months so I was catching it up with the updates, stepping from 2602 to 2603 and then to 2604 last night. Since that update, when I run "Get-SolutionUpdate" or "Get-SolutionUpdateEnvironment" the command hangs for a few minutes and then eventually returns with something like the following:
get-solutionUpdate : A WebException occurred while sending a RestRequest. WebException.Status: SendFailure on
https://FQDN:4900/providers/Microsoft.Update.Admin/updateLocations?api-version=2022-08-01
+ CategoryInfo : ConnectionError: (:) [Get-SolutionUpdate], SendFailureException
+ FullyQualifiedErrorId : SendFailureException,Microsoft.AzureStack.Lcm.PowerShell.GetSolutionUpdateCmdlet
get-solutionUpdate : Object reference not set to an instance of an object.
+ CategoryInfo : InvalidOperation: (:) [Get-SolutionUpdate], NullReferenceException
+ FullyQualifiedErrorId : NullReferenceException,Microsoft.AzureStack.Lcm.PowerShell.GetSolutionUpdateCmdletI have tried https://github.com/Azure/AzureLocal-Supportability/blob/main/TSG/Update/Get-SolutionUpdate-GatewayTimeout.md to try resolve it as it seemed similar but it hasn't helped. Any idea what might be going on?
After getting a ticket in with Support, they confirmed that the SSL certificate that had been assigned to port 4900 had been set to one with a CN of "URP". Deleting the association and then assigning a cert that has a Server Authentication purpose with the FQDN of the node and repeating this on each node solved the issue.
For those not familiar with the netsh commands to do this, first confirm the certificate being used by getting the thumbprint with "netsh http show sslcert ipport=0.0.0.0:4900", and then "get-childItem Cert:\LocalMachine\My\<thumbprint>" which will show the CN of the Cert.
If it's not the FQDN of the node, find the correct certificate by using this command to return all certs with a Server Authentication purpose "Get-ChildItem Cert:\LocalMachine\My | Where-Object {$_.EnhancedKeyUsageList.ObjectId -contains "1.3.6.1.5.5.7.3.1" -and ($_.Extensions | Where-Object {$_.Oid.FriendlyName -eq "Certificate Template Information"})}". Note the thumbprint of the cert with the nodes FQDN.
Delete the current incorrect certificate associated with the service with "netsh http delete sslcert ipport=0.0.0.0:4900" and then run "netsh http add sslcert ipport=0.0.0.0:4900 certhash=<thumbprint from above> appid='{11111111-2222-3333-4444-555555555555}'". Restart the service with "Restart-Service "Azure Stack HCI Update Service"" and then try run the commands again. You might need to relaunch the Powershell session as it seems to cache results if you've already tried running those commands in that session
4 Replies
After getting a ticket in with Support, they confirmed that the SSL certificate that had been assigned to port 4900 had been set to one with a CN of "URP". Deleting the association and then assigning a cert that has a Server Authentication purpose with the FQDN of the node and repeating this on each node solved the issue.
For those not familiar with the netsh commands to do this, first confirm the certificate being used by getting the thumbprint with "netsh http show sslcert ipport=0.0.0.0:4900", and then "get-childItem Cert:\LocalMachine\My\<thumbprint>" which will show the CN of the Cert.
If it's not the FQDN of the node, find the correct certificate by using this command to return all certs with a Server Authentication purpose "Get-ChildItem Cert:\LocalMachine\My | Where-Object {$_.EnhancedKeyUsageList.ObjectId -contains "1.3.6.1.5.5.7.3.1" -and ($_.Extensions | Where-Object {$_.Oid.FriendlyName -eq "Certificate Template Information"})}". Note the thumbprint of the cert with the nodes FQDN.
Delete the current incorrect certificate associated with the service with "netsh http delete sslcert ipport=0.0.0.0:4900" and then run "netsh http add sslcert ipport=0.0.0.0:4900 certhash=<thumbprint from above> appid='{11111111-2222-3333-4444-555555555555}'". Restart the service with "Restart-Service "Azure Stack HCI Update Service"" and then try run the commands again. You might need to relaunch the Powershell session as it seems to cache results if you've already tried running those commands in that session
The SendFailure is important. It means the cmdlet cannot complete its HTTPS request to the clustered Azure Stack HCI Update Service on port 4900; the NullReferenceException is usually a follow-on because no response object was returned. Since the GatewayTimeout procedure did not help, run Get-ActionPlanInstances and confirm it responds quickly. If it stalls, move the Azure Stack HCI Orchestrator Service Cluster Group. Next confirm the Azure Stack HCI Update Service Cluster Group is Online, note its owner node, and test DNS resolution and TCP 4900 to the reported FQDN from every node. On the owner, review the Application log for AzureStack Agent Lifecycle Agent events, especially Update Service termination or memory-limit errors. Move the Update Service cluster group once and retest. If the endpoint still fails, collect diagnostic logs and open Microsoft support with UTC timestamps; 2604 release notes do not list this exact error as a known issue.
Unfortunately, there are no relevant events that I can find in any event log. I've tried moving the service to either node but I still get the same results.
The plan is to make a ticket with Microsoft but at the moment my support plan seems to have dropped off and I'm stuck with a Basic support plan that doesn't allow me to create a ticket at the moment, at least until our vendor sorts out the support plan.
After updating to build 2604, the Get-SolutionUpdate and Get-SolutionUpdateEnvironment cmdlets can fail with SendFailure or NullReferenceException errors if the update orchestrator or cluster update resources are unhealthy, learn.microsoft.com/en-us/azure/azure-local/update/update-troubleshooting-23h2?view=azloc-2606
