Hi all,
We recently updated SCVMM 2019 to SCVMM 2019 UR2. The update ran smoothly, and SCVMM was back up and running. As a post-update task the SCVMM host agents of the HyperV hosts managed by SCVMM have to be updated. We have done this during previous updates and upgrades and never before ran into an issue.
In this particular case, we did run into an issue, and it had a major impact. After updating the agent on a single host, and noticing the update was applied correctly, we started applying the update to multiple hosts simultaneously (all running WS2019 DTC). Suddenly we noticed the loss of connectivity to vm's on 2 of our Hyper-V clusters. 1 hosts in cluster 1 and 2 host in cluster 2 failed the SCVMM agent update job, and also failed to drain themselves of their running virtual machines. Closer inspection showed vmms (Virtual Machine Management Service) was stuck in the Stop Pending state on all 3 hosts. After manually killing the service, it started back up, and the vm's eventually became available again.
We soon noticed that all affected vm's had suffered an unexpected shutdown, which resulted in considerable time spent to check if they were still functioning correctly. In some cases they weren't, and we had manually fix the application or resort to a restore.
Now we have already decided for ourselves that in future updates/upgrades we will drain hosts before attempting to update their SCVMM agent. I am however left wondering if this is a new issue, or if it was known before, and we somehow overlooked it in the countless times we read through the SCVMM update/upgrade manual.
