ATP sensor on Server 2016 DC crashing.

[Tony escamilla]

Currently we went through  a process of upgrading our DCs to 2016 after doing this and going through process of installing ATP sensor on the servers we are having the service crash or stay in a starting state. 

I get the following errors as well.

Application log

Event ID 1008

The Open Procedure for service "WmiApRpl" in DLL "C:\Windows\system32\wbem\wmiaprpl.dll" failed. Performance data for this service will not be available. The first four bytes (DWORD) of the Data section contains the error code.

The Open Procedure for service "Lsa" in DLL "C:\Windows\System32\Secur32.dll" failed. Performance data for this service will not be available. The first four bytes (DWORD) of the Data section contains the error code.

Event ID 2004

Unable to open the Server service performance object. The first four bytes (DWORD) of the Data section contains the status code.

System Log

Event ID 7032

The Service Control Manager tried to take a corrective action (Restart the service) after the unexpected termination of the Azure Advanced Threat Protection Sensor service, but this action failed with the following error:
The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

Event Id 7031

The Azure Advanced Threat Protection Sensor service terminated unexpectedly. It has done this 385 time(s). The following corrective action will be taken in 5000 milliseconds: Restart the service.

I can't seem to find anything regarding this or if I am missing something.  The prereq doesn't seem to show anything special missing so i am stumped.   Any help is welcome.

Thanks,

[#2]

@Tony escamilla 

Hi Tony,

Can you share the Log files for troubleshooting?

https://docs.microsoft.com/en-us/azure-advanced-threat-protection/troubleshooting-atp-using-logs

[#3]

@Tony escamilla , Also, can you make sure the Pla service is running correctly on the machine?

[#4]

I have had this issue pop up recently. Lots of 1008 errors and the ATP sensor wouldn't start and would error in the ATP portal for no communication. Server 2012R2

1008, Perflib

The Open Procedure for service "WmiApRpl" in DLL "C:\Windows\system32\wbem\wmiaprpl.dll" failed. Performance data for this service will not be available. The first four bytes (DWORD) of the Data section contains the error code.

The Open Procedure for service "Lsa" in DLL "C:\Windows\System32\Secur32.dll" failed. Performance data for this service will not be available. The first four bytes (DWORD) of the Data section contains the error code.

The Open Procedure for service ".NETFramework" in DLL "C:\Windows\system32\mscoree.dll" failed. Performance data for this service will not be available. The first four bytes (DWORD) of the Data section contains the error code.

Looking in the log located at "C:\Program Files\Azure Advanced Threat Protection Sensor\2.105.7563.11519\Logs\Microsoft.Tri.Sensor.log" I see this error

2020-01-22 12:31:49.1675 Warn PcapLibraryHelper Verify [Packet.dll-ProductName=WinPcap Packet.dll-ProductVersion=4.1.0.2980 wpcap.dll-ProductName=WinPcap wpcap.dll-ProductVersion=4.1.0.2980]
2020-01-22 12:31:49.4018 Error WinPcapDeviceList SharpPcap.PcapException: No interfaces found! Make sure libpcap/WinPcap is properly installed on the local machine.
at List<WinPcapDevice> SharpPcap.WinPcap.WinPcapDeviceList.Devices(string rpcapString, RemoteAuthentication remoteAuthentication)
at void SharpPcap.WinPcap.WinPcapDeviceList.Refresh()
at WinPcapDeviceList SharpPcap.WinPcap.WinPcapDeviceList.get_Instance()
at new Microsoft.Tri.Sensor.NetworkListener(IBufferPool bufferPool, IConfigurationManager configurationManager, IMetricManager metricManager, INetworkAdaptersManager networkAdaptersManager, IParsingOrchestrator parsingOrchestrator, IWorkspaceApplicationSensorApiJsonProxy workspaceApplicationSensorApiJsonProxy)
at object lambda_method(Closure, object[])
at object Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
at void Microsoft.Tri.Infrastructure.ModuleManager.AddModules(Type[] moduleTypes)
at new Microsoft.Tri.Sensor.SensorModuleManager()
at ModuleManager Microsoft.Tri.Sensor.SensorService.CreateModuleManager()
at async Task Microsoft.Tri.Infrastructure.Service.OnStartAsync()
at void Microsoft.Tri.Infrastructure.TaskExtension.Await(Task task)
at void Microsoft.Tri.Infrastructure.Service.OnStart(string[] args)

I downloaded and installed npcap, restarted the sensor and everything cleared up. 

Is there an issue with the ATP sensors install of WinPcap?

[#5]

@jomalin , the sensor should work fine with both most of the time.

Currently the default is winpcap, and if you require nic teaming support then we need npcap,

but you can work with npcap just fine even without teaming.

We are also considering at some point to make npcap the default and not winpcap.

As to why winpcap did not work for your case, it's hard to tell, usually it means there is another product installed that is also using winpcap but with a configuration we do not support.

If you would like to research it a support call might be in order, but if npcap just work for you, I guess that would be a waste of your time...

Eli

[#6]

@Eli Ofek 

Wireshark was also installed on the machines, I figured that was where the issue may have come from. All is well now. Thanks!

[#7]

@Eli Ofek 

I actually get the same errors as @jomalin but with one exception on these servers and it is mixture as some are 2012 R2 and others are 2016.  I also never have installed WinPcap.  I wasn't planning on installing npcap either as it doesn't look like it requires it unless i have a physical server and require NIC teaming. In my case all the servers (4) are running on VMware so the only requirement is the change on the NIC adapter being used.  As for npcap it seems that my installs work fine until the update process goes out and looks for updates and that is where all hell breaks loose.  the ATP sensor process stops and then never comes back on.  NpCap then gets installed which it seems is coming from the sensor but I have not been able to trace where that install is coming from.  What makes it even more weird is that it is only affecting 4 out of the more than 10 DCs we have in our environment.  Should I install Winncap to see if all works fine then. I do have a support ticket opened with support but we are just stumped as to why npcap is just being installed on these and not the rest. 

[#8]

@Tony escamilla  I am not aware of any code in the product that is installing npcap automatically.

For now the only option I know is deploying it manually.

As for why it fails, it depends on the out put in the logs.

If you have a support ticket open already then they should be able to tell why the failure is happening.

But I don't think you will find that AATP automatically installed npcap...

[#9]

@Eli Ofek 

One thing to note so i did a complete new install of the sensor.  There was no Npcap or winpcap or wireshark installed on the system.  It worked fine initially.  about an hour later looks to be the updater service kicks in and right around the same time npcap 0.9982 gets installed and these errors begin to happen.    Now i have also experimented with me manually installing npcap but same exact issues happened.  The sensor doesn't like it.   

Here is some info from the logs 

Microsoft.tri.sensor.updater.log:

2020-01-22 22:48:48.2754 Error ServiceControllerExtension ChangeServiceStatus failed to change service status [name=AATPSensor status=Running Exception=System.ServiceProcess.TimeoutException: Time out has expired and the operation has not been completed.
at System.ServiceProcess.ServiceController.WaitForStatus(ServiceControllerStatus desiredStatus, TimeSpan timeout)
at Microsoft.Tri.Infrastructure.ServiceControllerExtension.ChangeServiceStatus(String name, ServiceControllerStatus status, TimeSpan timeout, Nullable`1 awaitedStatus)]
2020-01-22 22:48:48.2754 Debug SoftwareUpdater RunTaskAsync Task completed [name=CheckSoftwareUpdatesAsync Elapsed=00:01:00.2710422]
2020-01-22 22:50:48.7122 Error ServiceControllerExtension ChangeServiceStatus failed to change service status [name=AATPSensor status=Running Exception=System.ServiceProcess.TimeoutException: Time out has expired and the operation has not been completed.
at System.ServiceProcess.ServiceController.WaitForStatus(ServiceControllerStatus desiredStatus, TimeSpan timeout)
at Microsoft.Tri.Infrastructure.ServiceControllerExtension.ChangeServiceStatus(String name, ServiceControllerStatus status, TimeSpan timeout, Nullable`1 awaitedStatus)]
2020-01-22 22:50:48.7122 Debug SoftwareUpdater RunTaskAsync Task completed [name=CheckSoftwareUpdatesAsync Elapsed=00:01:00.4341168]
2020-01-22 22:52:49.0582 Error ServiceControllerExtension ChangeServiceStatus failed to change service status [name=AATPSensor status=Running Exception=System.ServiceProcess.TimeoutException: Time out has expired and the operation has not been completed.
at System.ServiceProcess.ServiceController.WaitForStatus(ServiceControllerStatus desiredStatus, TimeSpan timeout)
at Microsoft.Tri.Infrastructure.ServiceControllerExtension.ChangeServiceStatus(String name, ServiceControllerStatus status, TimeSpan timeout, Nullable`1 awaitedStatus)]
2020-01-22 22:52:49.0582 Debug SoftwareUpdater RunTaskAsync Task completed [name=CheckSoftwareUpdatesAsync Elapsed=00:01:00.3380506]

Microsoft.tri.sensor-errors.log

2020-01-22 22:47:26.5764 Error FrameReader`1 CaptureFrames exception, exiting
Microsoft.Tri.Sensor.FrameReaderException: Failed reading frame [resultCode=-1 message=read error: PacketReceivePacket failed]
at bool Microsoft.Tri.Sensor.FrameReader<TCaptureDevice>.TryReadFrame(out DateTime time, out BufferSlice bufferSlice)
at bool Microsoft.Tri.Sensor.NetworkListener.ParseFrame(FrameReader frameReader)
at void Microsoft.Tri.Sensor.NetworkListener.CaptureFrames(LiveFrameReader[] liveFrameReaders)
2020-01-22 22:47:47.3509 Error WinPcapDeviceList SharpPcap.PcapException: No interfaces found! Make sure libpcap/WinPcap is properly installed on the local machine.
at List<WinPcapDevice> SharpPcap.WinPcap.WinPcapDeviceList.Devices(string rpcapString, RemoteAuthentication remoteAuthentication)
at void SharpPcap.WinPcap.WinPcapDeviceList.Refresh()
at WinPcapDeviceList SharpPcap.WinPcap.WinPcapDeviceList.get_Instance()
at new Microsoft.Tri.Sensor.NetworkListener(IBufferPool bufferPool, IConfigurationManager configurationManager, IMetricManager metricManager, INetworkAdaptersManager networkAdaptersManager, IParsingOrchestrator parsingOrchestrator, IWorkspaceApplicationSensorApiJsonProxy workspaceApplicationSensorApiJsonProxy)
at object lambda_method(Closure, object[])
at object Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
at void Microsoft.Tri.Infrastructure.ModuleManager.AddModules(Type[] moduleTypes)
at new Microsoft.Tri.Sensor.SensorModuleManager()
at ModuleManager Microsoft.Tri.Sensor.SensorService.CreateModuleManager()
at async Task Microsoft.Tri.Infrastructure.Service.OnStartAsync()
at void Microsoft.Tri.Infrastructure.TaskExtension.Await(Task task)
at void Microsoft.Tri.Infrastructure.Service.OnStart(string[] args)
2020-01-22 22:48:02.9984 Error WinPcapDeviceList SharpPcap.PcapException: No interfaces found! Make sure libpcap/WinPcap is properly installed on the local machine.

any ideas. 

[#10]

@Tony escamilla 

Can you share in a private message:

What is the workspace ID ?

What is the machine name?

What version of the sensor package are you using ? is it the latest?

I want to focus on it and see what telemetries it is sending.

Does it also happen on a fresh machine or only on this one?

The updater service kicks in every 30 sec or so, and only does something if it finds a new version at the back end, which happens mostly once a week unless we need to patch something quickly, so unless  you are using an old package, the updater should not really do anything post running for the first time until we really release a new version. (newest today is 2.106)

[#11]

@Eli Ofek

Thanks for the assist. I guess to help others.  In my case a 3rd party app was pushing out Nmap and along with it came Npcap.  I hadn't noticed because it wasn't by the typical methods of installs we utilize. As well as there was no trace of Nmap itself.  Once I ran ProcMon once again with some slight modifications to the filter as suggested by @Eli Ofek   I was able to find the culprit and fix my issue.  The problem is now gone and I have documentation for historical sake. 

Thanks again.