First published on MSDN on Mar 08, 2019
We
have
bec
o
me aware of a Windows Defender regression that c
an
block Service Fabric cluster upgrades.
Certain Windows Defender signatures contain a regression that can potentially hang Windows Task Scheduler
blocking
Windows applications that rely on Task Scheduler.
You can learn more about
this
regression in this
post
.
Impacted Customers:
This regression will only impact
Service Fabric cluster upgrades for
customers running Windows Server 2016.
Other OS images and Service Application upgrades are not affected as part of this regression.
On
a machine that is impacted by the regression, Windows Task scheduler would be in hung state
.
A
ny type of Service Fabric cluster upgrade would take down the Service Fabric node hosted on the machine permanently until an OS reboot / reimage happens.
Signatures that contain the regression have been rolled out broadly to most Azure machines running Windows Server 2016 + Windows Defender with a standard Defender signature update config. It is likely
that some
Windows Server 2016 machines Task Scheduler are already in hung state
.
If Task Scheduler is in hung state, a
ny Service Fabric cluster upgrade could bring down Service Fabric nodes and roll back due to node health policy violations.
Mitigation Steps:
If
you started a
Service Fabric cluster upgrade
which
brought down nodes permanently post rollback
. Please follow the steps below
:
On machines hosting down nodes,
Use
%
systemroot
%\System32\SchTasks.exe to verify if the Task Scheduler is in hung state. The command is supposed to return the list of scheduled tasks, if it does not return, then Task Scheduler is hung.
Reboot the machines and check if the nodes come back. If they don’t, reimage the machines.
If you cannot even RDP into the machines, directly reimage them.
SF runtime 6.3 support will be extended to March 31st, 2019.
