Azure ATP Sensor install failing (Updater Service do not start)

[PhilippFoeckeler]

Hello All!

We try to install the Azure ATP Sensor on a DC, setup wizard is running until this point

...then do some retries for about 3 minutes, during this time the service "Azure Advanced Threat Protection Sensor Updater" is several times on state "starting" und back to not started.

Then setup fails with 0x80070643 and do a rollback.

In the "Microsoft.Tri.Sensor.Updater-Errors" log, we find this error every 10 seconds during the setup:

2019-12-23 11:27:37.8384 Error CommunicationWebClient+<SendWithRetryAsync>d__8`1 Microsoft.Tri.Infrastructure.ExtendedException: Sanitized exception: [Type=System.Net.Http.HttpRequestExceptionMessage=7INzM3PVZQKggOiiHcWjqw==StackTrace= at async Task<HttpResponseMessage> System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task<HttpResponseMessage> sendTask, HttpRequestMessage request, CancellationTokenSource cts, bool disposeCts)
at async Task<TResponse> Microsoft.Tri.Common.CommunicationWebClient.SendAsync<TResponse>(byte[] requestBytes, int offset, int count)
at async Task<TResponse> Microsoft.Tri.Common.CommunicationWebClient.SendWithRetryAsync<TResponse>(byte[] requestBytes, int offset, int count)InnerException=Microsoft.Tri.Infrastructure.ExtendedException: Sanitized exception: [Type=System.Net.WebExceptionMessage=5iiWw0iPCPzCGdZStU4OxA==StackTrace= at Stream System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, out TransportContext context)
at void System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)InnerException=]]
at async Task<TResponse> Microsoft.Tri.Common.CommunicationWebClient.SendWithRetryAsync<TResponse>(byte[] requestBytes, int offset, int count)
at async Task<TResponse> Microsoft.Tri.Common.CommunicationWebClient.SendAsync<TResponse>(IRequestWithResponse<TResponse> request)
at async Task<TResponse> Microsoft.Tri.Sensor.Common.ServiceProxy<TWebClientConfiguration>.SendAsync<TResponse>(IRequestWithResponse<TResponse> request)
at async Task Microsoft.Tri.Sensor.Updater.SensorUpdaterConfigurationUpdater.UpdateConfigurationAsync(bool isStarted)
at void Microsoft.Tri.Infrastructure.TaskExtension.Await(Task task)
at new Microsoft.Tri.Sensor.Updater.SensorUpdaterConfigurationUpdater(IConfigurationManager configurationManager, IMetricManager metricManager, ISecretManager secretManager, IWorkspaceApplicationSensorApiJsonProxy workspaceApplicationSensorApiJsonProxy)
at object lambda_method(Closure, object[])
at object Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
at void Microsoft.Tri.Infrastructure.ModuleManager.AddModules(Type[] moduleTypes)
at ModuleManager Microsoft.Tri.Sensor.Updater.SensorUpdaterService.CreateModuleManager()
at async Task Microsoft.Tri.Infrastructure.Service.OnStartAsync()
at void Microsoft.Tri.Infrastructure.TaskExtension.Await(Task task)
at void Microsoft.Tri.Infrastructure.Service.OnStart(string[] args)

A proxy is used which allows access to *.atp.azure.com without auth. In proxy logs, we see no block for this server, only successful requests from this DC. There is no indication that 443 would be blocked somewhere else...

The AD account which is configured in the ATP portal was checked, domain is given in FQDN there and the password is correct.

Any ideas someone?

[#2]

@Vishal_Sharma_4224 

this is log erros

22B8:27F0][2020-06-02T12:20:23]i301: Applying execute package: MsiPackage, action: Install, path: C:\ProgramData\Package Cache\{F288B034-5037-4734-8CBD-B38388555443}v2.0.0.0\Microsoft.Tri.Sensor.Deployment.Package.msi, arguments: ' ARPSYSTEMCOMPONENT="1" MSIFASTINSTALL="7" ACCESSKEY="*****" InstallationPath="C:\Program Files\Azure Advanced Threat Protection Sensor" InstalledVersion="" PROXYCONFIGURATION="*****" WixBundleOriginalSourceFolder="C:\Sources\Azure ATP Sensor Setup\"'

[22B8:27F0][2020-06-02T12:21:09]e000: Error 0x80070643: Failed to install MSI package.

[22B8:27F0][2020-06-02T12:21:09]e000: Error 0x80070643: Failed to execute MSI package.

[229C:1AE8][2020-06-02T12:21:09]e000: Error 0x80070643: Failed to configure per-machine MSI package.

[229C:1AE8][2020-06-02T12:21:09]i319: Applied execute package: MsiPackage, result: 0x80070643, restart: None

[229C:1AE8][2020-06-02T12:21:09]e000: Error 0x80070643: Failed to execute MSI package.

[#3]

@Mina Kamel 

Please paste the sensor\deployer logs.. 

[#4]

@Vishal_Sharma_4224 

2020-06-02 09:21:00.9589 Info Program Main Deployer started [arguments=s6RF2dHzFWtQ6a0bV+YdnQ==]
2020-06-02 09:21:01.1129 Debug InstallActionGroup Apply started
2020-06-02 09:21:01.1139 Debug CreateCertificateAction Apply started [suppressFailure=False]
2020-06-02 09:21:02.1310 Debug CreateCertificateAction Apply finished
2020-06-02 09:21:02.1320 Debug CreateSensorAction Apply started [suppressFailure=False]
2020-06-02 09:21:04.7894 Debug CreateSensorAction Apply finished
2020-06-02 09:21:04.7894 Debug SaveSensorMandatoryConfigurationAction Apply started [suppressFailure=False]
2020-06-02 09:21:04.8462 Debug SaveSensorMandatoryConfigurationAction Apply finished
2020-06-02 09:21:04.8462 Debug CreateServicesActionGroup Apply started
2020-06-02 09:21:04.8462 Debug CreateServiceAction Apply started [suppressFailure=False]
2020-06-02 09:21:04.8942 Debug CreateServiceAction Apply finished
2020-06-02 09:21:04.8942 Debug SetServiceDescriptionAction Apply started [suppressFailure=False]
2020-06-02 09:21:04.9102 Debug SetServiceDescriptionAction Apply finished
2020-06-02 09:21:04.9102 Debug ConfigureServiceAction Apply started [suppressFailure=False]
2020-06-02 09:21:04.9312 Debug ConfigureServiceAction Apply finished
2020-06-02 09:21:04.9312 Debug SetServicePreshutdownTimeoutAction Apply started [suppressFailure=False]
2020-06-02 09:21:04.9472 Debug SetServicePreshutdownTimeoutAction Apply finished
2020-06-02 09:21:04.9472 Debug CreateServiceAction Apply started [suppressFailure=False]
2020-06-02 09:21:04.9652 Debug CreateServiceAction Apply finished
2020-06-02 09:21:04.9652 Debug SetServiceDescriptionAction Apply started [suppressFailure=False]
2020-06-02 09:21:04.9842 Debug SetServiceDescriptionAction Apply finished
2020-06-02 09:21:04.9842 Debug ConfigureServiceAction Apply started [suppressFailure=False]
2020-06-02 09:21:05.0032 Debug ConfigureServiceAction Apply finished
2020-06-02 09:21:05.0032 Debug SetServicePreshutdownTimeoutAction Apply started [suppressFailure=False]
2020-06-02 09:21:05.0222 Debug SetServicePreshutdownTimeoutAction Apply finished
2020-06-02 09:21:05.0222 Debug CreateServicesActionGroup Apply finished
2020-06-02 09:21:05.0222 Debug ConfigureVirtualServiceAccountAction Apply started [suppressFailure=False]
2020-06-02 09:21:05.2782 Debug ConfigureVirtualServiceAccountAction Apply finished
2020-06-02 09:21:05.2782 Debug InstallWinPcapAction Apply started [suppressFailure=False]
2020-06-02 09:21:07.3383 Debug ServiceControllerExtension CreateKernelDriver service already exists [name=npf]
2020-06-02 09:21:07.4733 Error ServiceControllerExtension ChangeServiceStatus failed to change service status [name=npf status=Running Exception=System.InvalidOperationException: Cannot start service npf on computer '.'. ---> System.ComponentModel.Win32Exception: The system cannot find the file specified
--- End of inner exception stack trace ---
at System.ServiceProcess.ServiceController.Start(String[] args)
at Microsoft.Tri.Infrastructure.ServiceControllerExtension.ChangeServiceStatus(String name, ServiceControllerStatus status, TimeSpan timeout, Nullable`1 awaitedStatus)]
2020-06-02 09:21:07.4783 Debug InstallActionGroup Revert started
2020-06-02 09:21:07.4793 Warn InstallActionGroup Revert reverting [rollbackAction=ConfigureVirtualServiceAccountAction index=0 count=5]
2020-06-02 09:21:07.4793 Debug ConfigureVirtualServiceAccountAction Revert started
2020-06-02 09:21:07.4793 Debug ConfigureVirtualServiceAccountAction Revert finished
2020-06-02 09:21:07.4793 Warn InstallActionGroup Revert reverting [rollbackAction=CreateServicesActionGroup index=1 count=5]
2020-06-02 09:21:07.4793 Debug CreateServicesActionGroup Revert started
2020-06-02 09:21:07.4793 Warn CreateServicesActionGroup Revert reverting [rollbackAction=SetServicePreshutdownTimeoutAction index=0 count=8]
2020-06-02 09:21:07.4793 Debug SetServicePreshutdownTimeoutAction Revert started
2020-06-02 09:21:07.4793 Debug SetServicePreshutdownTimeoutAction Revert finished
2020-06-02 09:21:07.4793 Warn CreateServicesActionGroup Revert reverting [rollbackAction=ConfigureServiceAction index=1 count=8]
2020-06-02 09:21:07.4793 Debug ConfigureServiceAction Revert started
2020-06-02 09:21:07.4793 Debug ConfigureServiceAction Revert finished
2020-06-02 09:21:07.4793 Warn CreateServicesActionGroup Revert reverting [rollbackAction=SetServiceDescriptionAction index=2 count=8]
2020-06-02 09:21:07.4793 Debug SetServiceDescriptionAction Revert started
2020-06-02 09:21:07.4793 Debug SetServiceDescriptionAction Revert finished
2020-06-02 09:21:07.4793 Warn CreateServicesActionGroup Revert reverting [rollbackAction=CreateServiceAction index=3 count=8]
2020-06-02 09:21:07.4793 Debug CreateServiceAction Revert started
2020-06-02 09:21:07.5253 Debug ServiceControllerExtension DeleteService succeeded [name=AATPSensor]
2020-06-02 09:21:07.5253 Debug CreateServiceAction Revert finished
2020-06-02 09:21:07.5253 Warn CreateServicesActionGroup Revert reverting [rollbackAction=SetServicePreshutdownTimeoutAction index=4 count=8]
2020-06-02 09:21:07.5253 Debug SetServicePreshutdownTimeoutAction Revert started
2020-06-02 09:21:07.5253 Debug SetServicePreshutdownTimeoutAction Revert finished
2020-06-02 09:21:07.5253 Warn CreateServicesActionGroup Revert reverting [rollbackAction=ConfigureServiceAction index=5 count=8]
2020-06-02 09:21:07.5253 Debug ConfigureServiceAction Revert started
2020-06-02 09:21:07.5253 Debug ConfigureServiceAction Revert finished
2020-06-02 09:21:07.5253 Warn CreateServicesActionGroup Revert reverting [rollbackAction=SetServiceDescriptionAction index=6 count=8]
2020-06-02 09:21:07.5253 Debug SetServiceDescriptionAction Revert started
2020-06-02 09:21:07.5253 Debug SetServiceDescriptionAction Revert finished
2020-06-02 09:21:07.5253 Warn CreateServicesActionGroup Revert reverting [rollbackAction=CreateServiceAction index=7 count=8]
2020-06-02 09:21:07.5253 Debug CreateServiceAction Revert started
2020-06-02 09:21:07.5743 Debug ServiceControllerExtension DeleteService succeeded [name=AATPSensorUpdater]
2020-06-02 09:21:07.5753 Debug CreateServiceAction Revert finished
2020-06-02 09:21:07.5753 Debug CreateServicesActionGroup Revert finished
2020-06-02 09:21:07.5753 Warn InstallActionGroup Revert reverting [rollbackAction=SaveSensorMandatoryConfigurationAction index=2 count=5]
2020-06-02 09:21:07.5753 Debug SaveSensorMandatoryConfigurationAction Revert started
2020-06-02 09:21:07.5753 Debug SaveSensorMandatoryConfigurationAction Revert finished
2020-06-02 09:21:07.5753 Warn InstallActionGroup Revert reverting [rollbackAction=CreateSensorAction index=3 count=5]
2020-06-02 09:21:07.5753 Debug CreateSensorAction Revert started
2020-06-02 09:21:08.6974 Debug CreateSensorAction Revert finished
2020-06-02 09:21:08.6974 Warn InstallActionGroup Revert reverting [rollbackAction=CreateCertificateAction index=4 count=5]
2020-06-02 09:21:08.6984 Debug CreateCertificateAction Revert started
2020-06-02 09:21:08.7234 Debug CreateCertificateAction Revert finished
2020-06-02 09:21:08.7234 Debug InstallActionGroup Revert finished
2020-06-02 09:21:08.9814 Error DeploymentAction Deployer failed
Microsoft.Tri.Infrastructure.ExtendedException: Apply failed [Type=InstallWinPcapAction]
at void Microsoft.Tri.Sensor.Common.DeploymentAction.Apply(bool suppressFailure)
at void Microsoft.Tri.Sensor.Common.DeploymentActionGroup.Apply(bool suppressFailure)
at int Microsoft.Tri.Sensor.Deployment.Deployer.Program.Main(string[] commandLineArguments)

[#5]

@Mina Kamel is NIC Teaming enabled on this machine..

In this case, the NPF service is not starting up, which works hand in hand with Winpcap\Npcap to capture the network activities.

Which Server OS are you running with? 

I 'd suggest you to install npcap which can be downloaded with the help link below (Remove if you are using any other Npcap version than 0.9984)
https://nmap.org/npcap/dist/npcap-0.9984.exe

Then,try steps below in the sequential manner:

1. Verify current files, Npcap & Npf services exist

In PowerShell:

get-item C:\Windows\System32\drivers\npf.sys

get-item C:\Windows\System32\drivers\npcap.sys

2. Verify service exist

Get-Service Npf | FL

Get-Service Npcap | FL

If npf service is still not started, reboot the sever. 

3. Install Azure ATP Sensor

Then Try & install the sensor setup .

[#6]

@Vishal_Sharma_4224  thanks for supporting me

on the physical DC the NIC Teaming it was configured before, but now is not configured,

Which Server OS are you running with? 

i have two DCs ( one Physical and one VM on VMWare) both are WS 2012 R2 standard, with DHCP role installed.

 i am trying to install NPCAP but i am facing an error 

on the physical server 

Npf service is running and no  Npcap  service.

on VM Server

Npf service is stopping and no  Npcap  service.

[#7]

Download the Npcap version 0.9984 installer from https://nmap.org/npcap/.
Alternatively, request the OEM version of the Npcap driver (that supports silent installation) from the support team.
Copies of Npcap do not count towards the five copy, five computer or fiver user licensing limitation if they are installed and used solely in conjunction with Azure ATP. For more information, see NPCAP licensing.
If you have not yet installed the sensor:

Uninstall WinPcap, if it was installed.
Install Npcap with the following options: loopback_support=no & winpcap_mode=yes.
If using the GUI installer, deselect the loopback support and select WinPcap mode.
Install the sensor package.
If you already installed the sensor:

Uninstall the sensor.
Uninstall WinPcap.
Install Npcap with the following options: loopback_support=no & winpcap_mode=yes
If using the GUI installer, deselect the loopback support and select WinPcap mode.
Reinstall the sensor package.

[#8]

thanks Lewis, i have issue while installing NPCAP 0.9984

[#9]

@Mina Kamel  if npcap is failing but you see npf service is running, that means you most likely have winpcap installed on the machine.

try uninstalling it, and if there is no reference to it, try to stop and delete the npf service:

sc stop npf

sc delete npf

and try the npcap install again.

If it still fails, open a support case, we can escalate this to npcap support, and we will need the full npcap install logs...

[#10]

@Eli Ofek 

Thanks for your help, I have followed your steps with no Success, i have opened support case with MS, but support engineer keep sending steps which i had followed before.

[#11]

@Mina Kamel  1st tier support need to cover all the basics before they can escalate...
What is the ticket number ?

[#12]

Support request number: 120060224002961

[#13]

@Mina Kamel , I reviewed the case, it turns out that you actually got bumped directly to the best regional escalation engineer.

He will still need to cover the basic before he can escalate further.
While you already did most of the documented steps and some that were proposed here, 
Most customers don't so support are forced to make sure everything was covered correctly before raising it more. I can understand this could be a bit frustrating, please bare with it. 
This way makes sure you are on the right track to resolution.

Once all the basics are covered and all data collected, it will be ready to be escalated to AATP Engineering. and/or npcap support.
You are in good hands.

Eli

[#14]

Hi @PhilippFoeckeler 

Try to execute the steps described here: https://docs.microsoft.com/en-us/azure-advanced-threat-protection/configure-proxy#configure-the-prox...

[#15 · Accepted solution]

Actually, the solution in our case was to use Silent Installation (https://docs.microsoft.com/en-us/azure-advanced-threat-protection/atp-silent-installation#proxy-auth...) and provide the Proxy Information in the commandline.

Thank you all for helping and advising!!!

[#16]

@Vishal_Sharma_4224 Thanks, worked like a charm.

[#17]

After 5 hours of troubleshooting, we found a solution to our problem beyond the steps listed below. The solution involved removing the account (gMSA in our case) from the Directory Services accounts under security.microsoft.com > Settings > Identities > Directory Services accounts. Once we re-added the account, the sensor "service status" changed to "running" and the sensor status to "up to date."

Cause of the problem: The update from the Azure ATP Sensor likely went wrong.

Steps Taken:

1- Disabled Services: We disabled both the "Azure Advanced Threat Protection Sensor Updater" and "Azure Advanced Threat Protection Sensor" services.
2- Software Removal: We removed the Azure ATP sensor software from the affected Domain Controller (DC). We encountered difficulties during this process and had to use the Microsoft support article https://support.microsoft.com/en-gb/topic/fix-problems-that-block-programs-from-being-installed-or-r... to resolve them before running the uninstall program from Windows itself.
3- File Deletion: We deleted all files located in the "C:\Program Files\Azure Advanced Threat Protection Sensor" directory.
4- Service Removal (Command Prompt): As administrator, we ran the following commands in a command prompt window to remove the services:
sc delete AATPSensor
sc delete AATPSensorUpdater
5- Server Reboot: We rebooted the server.
6- Readiness Script: We ran the script provided by Microsoft https://github.com/microsoft/Microsoft-Defender-for-Identity (all checks resulted in "OK").
7- New Sensor Deployment: We returned to the Microsoft portal and added a new sensor. We ensured we used the same access key from the downloaded installer.
8- IPv4 TSO Offload: Since we were working with a virtual machine (VM), we disabled IPv4 TCP Segmentation Offloading (TSO) as recommended in the Microsoft documentation https://learn.microsoft.com/en-us/connectors/wdatp/.
9- gMSA Account Removal and Re-addition: We removed and then re-added the gMSA account under [invalid URL removed] > Settings > Identities > Directory Services accounts.

Following these steps, everything functioned correctly, and the service was running.

Additional Tip: Don't overlook your log files! Check for relevant information in "C:\Program Files\Azure Advanced Threat Protection Sensor\VersionInUse\Logs"

I hope this helps someone else out!