AATPSensor stuck in start pending and AATPSensorUpdater logging errors

[AndreBaker]

We recently installed a number of VMWare VM W2019 domain controllers without GUI.

AATPSensor 2.178.15200.33528 was installed like so:

"Azure ATP sensor Setup.exe" /quiet NetFrameworkCommandLineArguments="/q" AccessKey="abc_etc"

After installation we are seeing that the AATPSensor is stuck in start pending and every 6 minutes we see that AATPSensorUpdater logs the following:

System.ServiceProcess.TimeoutException: Time out has expired and the operation has not been completed.

It seems like the AATPSensorUpdater is stuck in a loop because the AATPSensor is perpetually on start pending and never actually starts.

We have this issue on a handful of VMWare VM W2019 domain controllers without GUI, but not all domain controllers.

We already tried the usual remove - reboot - install for the Azure ATP sensor Setup.

Do you have any hints where we could look to try and get this resolved? Perhaps more verbose logging to find out why or where the AATPSensor service get stuck on at startup?

The last 10 lines in Microsoft.Tri.Sensor.log:

2022-05-05 14:00:42.7162 Debug NetworkAdaptersManager SetState Creating
2022-05-05 14:00:42.7631 Debug NetworkAdaptersManager UpdateIpAddresses ignoring network traffic [ignoredNetworkAdapters= _ignoredIpAddresses=]
2022-05-05 14:00:42.7631 Debug NetworkActivityEntityResolver SetState Creating
2022-05-05 14:00:42.8412 Debug DroppingPredictionManager SetState Creating
2022-05-05 14:00:43.1694 Debug BufferPool SetState Creating
2022-05-05 14:00:43.2631 Debug ParsingOrchestrator SetState Creating
2022-05-05 14:00:43.5288 Debug Parser Initialize AdwsParser.IsEnabled=False
2022-05-05 14:00:43.6225 Debug Parser Initialize AdwsParser.IsEnabled=False
2022-05-05 14:00:43.6381 Debug NetworkListener SetState Creating
2022-05-05 14:00:43.6381 Warn PcapLibraryHelper Verify [pcapProductLibrary=Npcap pcapProductVersion=1.0]

The last 10 lines in Microsoft.Tri.Sensor.Updater.log:

2022-05-05 14:25:24.8350 Error ServiceControllerExtension ChangeServiceStatus failed to change service status [name=AATPSensor status=Running Exception=System.ServiceProcess.TimeoutException: Time out has expired and the operation has not been completed.
at System.ServiceProcess.ServiceController.WaitForStatus(ServiceControllerStatus desiredStatus, TimeSpan timeout)
at Microsoft.Tri.Infrastructure.ServiceControllerExtension.ChangeServiceStatus(String name, ServiceControllerStatus status, TimeSpan timeout, Nullable`1 awaitedStatus)]
2022-05-05 14:25:24.8350 Debug SoftwareUpdater RunTaskAsync Task completed [name=CheckSoftwareUpdatesAsync Elapsed=00:01:00.3078576]
2022-05-05 14:31:24.9962 Error ServiceControllerExtension ChangeServiceStatus failed to change service status [name=AATPSensor status=Running Exception=System.ServiceProcess.TimeoutException: Time out has expired and the operation has not been completed.
at System.ServiceProcess.ServiceController.WaitForStatus(ServiceControllerStatus desiredStatus, TimeSpan timeout)
at Microsoft.Tri.Infrastructure.ServiceControllerExtension.ChangeServiceStatus(String name, ServiceControllerStatus status, TimeSpan timeout, Nullable`1 awaitedStatus)]
2022-05-05 14:31:24.9962 Debug SoftwareUpdater RunTaskAsync Task completed [name=CheckSoftwareUpdatesAsync Elapsed=00:01:00.1463938]
2022-05-05 14:37:25.1352 Error ServiceControllerExtension ChangeServiceStatus failed to change service status [name=AATPSensor status=Running Exception=System.ServiceProcess.TimeoutException: Time out has expired and the operation has not been completed.
at System.ServiceProcess.ServiceController.WaitForStatus(ServiceControllerStatus desiredStatus, TimeSpan timeout)
at Microsoft.Tri.Infrastructure.ServiceControllerExtension.ChangeServiceStatus(String name, ServiceControllerStatus status, TimeSpan timeout, Nullable`1 awaitedStatus)]
2022-05-05 14:37:25.1352 Debug SoftwareUpdater RunTaskAsync Task completed [name=CheckSoftwareUpdatesAsync Elapsed=00:01:00.1247188]

The last 10 lines in Microsoft.Tri.Sensor.Updater-Errors.log:

2022-05-05 14:25:24.8350 Error ServiceControllerExtension ChangeServiceStatus failed to change service status [name=AATPSensor status=Running Exception=System.ServiceProcess.TimeoutException: Time out has expired and the operation has not been completed.
at System.ServiceProcess.ServiceController.WaitForStatus(ServiceControllerStatus desiredStatus, TimeSpan timeout)
at Microsoft.Tri.Infrastructure.ServiceControllerExtension.ChangeServiceStatus(String name, ServiceControllerStatus status, TimeSpan timeout, Nullable`1 awaitedStatus)]
2022-05-05 14:31:24.9962 Error ServiceControllerExtension ChangeServiceStatus failed to change service status [name=AATPSensor status=Running Exception=System.ServiceProcess.TimeoutException: Time out has expired and the operation has not been completed.
at System.ServiceProcess.ServiceController.WaitForStatus(ServiceControllerStatus desiredStatus, TimeSpan timeout)
at Microsoft.Tri.Infrastructure.ServiceControllerExtension.ChangeServiceStatus(String name, ServiceControllerStatus status, TimeSpan timeout, Nullable`1 awaitedStatus)]
2022-05-05 14:37:25.1352 Error ServiceControllerExtension ChangeServiceStatus failed to change service status [name=AATPSensor status=Running Exception=System.ServiceProcess.TimeoutException: Time out has expired and the operation has not been completed.
at System.ServiceProcess.ServiceController.WaitForStatus(ServiceControllerStatus desiredStatus, TimeSpan timeout)
at Microsoft.Tri.Infrastructure.ServiceControllerExtension.ChangeServiceStatus(String name, ServiceControllerStatus status, TimeSpan timeout, Nullable`1 awaitedStatus)]
2022-05-05 14:43:25.4437 Error ServiceControllerExtension ChangeServiceStatus failed to change service status [name=AATPSensor status=Running Exception=System.ServiceProcess.TimeoutException: Time out has expired and the operation has not been completed.
at System.ServiceProcess.ServiceController.WaitForStatus(ServiceControllerStatus desiredStatus, TimeSpan timeout)
at Microsoft.Tri.Infrastructure.ServiceControllerExtension.ChangeServiceStatus(String name, ServiceControllerStatus status, TimeSpan timeout, Nullable`1 awaitedStatus)]

[#2 · Accepted solution]

@AndreBaker The sensor log always get stuck on the line with "PcapLibraryHelper Verify" ?
What happens if you kill the sensor process from task manager? does it restart the sequence and gets stuck again on the exact same line ?

If yes, most likely Npcap was installed on the machine with /admin_only option, and we don’t support it.
uninstall npcap, and reinstall with the supported attributes.
https://docs.microsoft.com/en-us/defender-for-identity/technical-faq#how-do-i-download-and-install-t...

[#3]

@Eli Ofek Yes, the sensor log always get stuck at this line when we kill the service and it starts again.

Warn PcapLibraryHelper Verify [pcapProductLibrary=Npcap pcapProductVersion=1.0]

We uninstalled npcap:

"c:\Program Files\Npcap\Uninstall.exe" /S

Then installed with the recommended params:

npcap\npcap-1.00-oem.exe /S /loopback_support=no /winpcap_mode=yes

Then terminated the Microsoft.Tri.Sensor.exe & Microsoft.Tri.Sensor.Updater.exe service processes.

Now the AATPSensor and AATPSensorUpdater are both running without issues and are showing up in the ATP sensors tab as running as well.

Thanks a lot for assisting!