ATP sensor service failed to start

[Hjuanola]

Hello everyone,

I've tried to deploy an ATP sensor on a server and the AATPSensor service can't seem to start properly (Starting status indefinitely) 

Error in the Windows Event Log :

 

Microsoft.Tri.Sensor

Spoiler

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Tri.Infrastructure.TaskExtension.Await[TResult](Task`1 task)
at Microsoft.Tri.Infrastructure.VirtualizationTypeHelper.<GetVirtualizationType>g__IsUriAccessible|7_0(Uri uri, <>c__DisplayClass7_0& )
at Microsoft.Tri.Infrastructure.VirtualizationTypeHelper.GetVirtualizationType()
at Microsoft.Tri.Infrastructure.Machine.get_VirtualizationType()]
2022-12-16 14:54:21.0039 Debug ConfigurationManager SetState Creating
2022-12-16 14:54:21.0195 Debug SensorSecretManager SetState Creating
2022-12-16 14:54:21.0195 Debug WorkspaceApplicationSensorApiJsonProxy SetState Creating
2022-12-16 14:54:21.0664 Debug WorkspaceApplicationSensorApiProtobufProxy SetState Creating
2022-12-16 14:54:21.0820 Debug MetricManager SetState Creating
2022-12-16 14:54:22.0212 Debug SensorConfigurationUpdater SetState Creating
2022-12-16 14:54:22.0212 Debug SanitizationManager SetState Creating
2022-12-16 14:54:22.0212 Debug SensorUpdaterProxy SetState Creating
2022-12-16 14:54:22.0525 Debug SensorPerformanceCounterMetricManager SetState Creating
2022-12-16 14:54:22.0525 Debug SensorMetricSender SetState Creating
2022-12-16 14:54:22.0525 Debug SensorAppDomainManager SetState Creating
2022-12-16 14:54:22.0525 Debug SensorTelemetrySender SetState Creating
2022-12-16 14:54:22.0681 Debug SensorTraceManager SetState Creating
2022-12-16 14:54:22.0837 Debug SensorExceptionStatisticsManager SetState Creating
2022-12-16 14:54:22.0837 Debug SensorResourceManager SetState Creating
2022-12-16 14:54:32.1742 Debug SyslogClient SetState Creating
2022-12-16 14:54:32.1742 Debug EntitySender SetState Creating
2022-12-16 14:54:32.2054 Debug FailedNetworkNameResolutionStatistics SetState Creating
2022-12-16 14:54:32.2054 Debug TcpClientWrapper SetState Creating
2022-12-16 14:54:32.2054 Debug UdpClientWrapper SetState Creating
2022-12-16 14:54:32.2210 Debug AfpClient SetState Creating
2022-12-16 14:54:32.2210 Debug DnsClient SetState Creating
2022-12-16 14:54:32.2367 Debug HttpTlsClient SetState Creating
2022-12-16 14:54:32.2367 Debug NetbiosClient SetState Creating
2022-12-16 14:54:32.2367 Debug RdpTlsClient SetState Creating
2022-12-16 14:54:32.2367 Debug RpcNtlmClient SetState Creating
2022-12-16 14:54:32.2367 Debug SmbNtlmClient SetState Creating
2022-12-16 14:54:32.2523 Debug NetworkNameResolver SetState Creating
2022-12-16 14:54:32.2523 Debug LocalImpersonationManager SetState Creating
2022-12-16 14:54:32.2523 Debug RemoteImpersonationManager SetState Creating
2022-12-16 14:54:32.2523 Debug DomainTrustMappingManager SetState Creating
2022-12-16 14:54:32.2679 Debug DirectoryServicesDomainNetworkCredentialsManager SetState Creating
2022-12-16 14:54:32.6272 Debug DirectoryServicesClient SetState Creating
2022-12-16 14:54:32.6897 Info RemoteImpersonationManager CreateImpersonatorInternalAsync started [UserName=gmsa Domain=domain1 IsGroupManagedServiceAccount=True]
2022-12-16 14:54:32.7053 Info RemoteImpersonationManager GetGroupManagedServiceAccountTokenAsync finished [UserName=gmsa Domain=domain1 IsSuccess=False]
2022-12-16 14:54:32.7053 Info RemoteImpersonationManager CreateImpersonatorInternalAsync finished [UserName=gmsa Domain=domain1]
2022-12-16 14:54:32.7053 Warn DirectoryServicesClient CreateLdapConnectionAsync failed to retrieve group managed service account password. [DomainControllerDnsName=EUR-NT-CTLPT.domain1 Domain=domain1 UserName=gmsa ]
2022-12-16 14:54:32.7053 Info RemoteImpersonationManager CreateImpersonatorInternalAsync started [UserName=gmsa Domain=domain2 IsGroupManagedServiceAccount=True]
2022-12-16 14:54:43.7517 Info RemoteImpersonationManager GetGroupManagedServiceAccountTokenAsync finished [UserName=gmsa Domain=domain2 IsSuccess=False]
2022-12-16 14:54:43.7517 Info RemoteImpersonationManager CreateImpersonatorInternalAsync finished [UserName=gmsa Domain=domain2]
2022-12-16 14:54:43.7517 Warn DirectoryServicesClient CreateLdapConnectionAsync failed to retrieve group managed service account password. [DomainControllerDnsName=EUR-NT-CTLPT.domain1 Domain=domain2 UserName=gmsa ]
2022-12-16 14:54:43.7517 Info RemoteImpersonationManager CreateImpersonatorInternalAsync started [UserName=gmsa Domain=domain3 IsGroupManagedServiceAccount=True]
2022-12-16 14:54:43.7517 Info RemoteImpersonationManager GetGroupManagedServiceAccountTokenAsync finished [UserName=gmsa Domain=domain3 IsSuccess=False]
2022-12-16 14:54:43.7517 Info RemoteImpersonationManager CreateImpersonatorInternalAsync finished [UserName=gmsa Domain=domain3]
2022-12-16 14:54:43.7517 Warn DirectoryServicesClient CreateLdapConnectionAsync failed to retrieve group managed service account password. [DomainControllerDnsName=EUR-NT-CTLPT.domain1 Domain=domain3 UserName=gmsa]
2022-12-16 14:54:44.3609 Error DirectoryServicesClient+<CreateLdapConnectionAsync>d__48 Microsoft.Tri.Infrastructure.ExtendedException: CreateLdapConnectionAsync failed [DomainControllerDnsName=EUR-NT-CTLPT.domain1]
at async Task<LdapConnection> Microsoft.Tri.Sensor.DirectoryServicesClient.CreateLdapConnectionAsync(DomainControllerConnectionData domainControllerConnectionData, bool isGlobalCatalog, bool isTraversing)
at async Task<bool> Microsoft.Tri.Sensor.DirectoryServicesClient.TryCreateLdapConnectionAsync(DomainControllerConnectionData domainControllerConnectionData, bool isGlobalCatalog, bool isTraversing)
2022-12-16 14:54:44.3765 Error DirectoryServicesClient Microsoft.Tri.Infrastructure.ExtendedException: Failed to communicate with configured domain controllers [ _domainControllerConnectionDatas=EUR-NT-CTLPT.domain1]
at new Microsoft.Tri.Sensor.DirectoryServicesClient(IConfigurationManager configurationManager, IDirectoryServicesDomainNetworkCredentialsManager domainNetworkCredentialsManager, IDomainTrustMappingManager domainTrustMappingManager, IRemoteImpersonationManager remoteImpersonationManager, IMetricManager metricManager, IWorkspaceApplicationSensorApiJsonProxy workspaceApplicationSensorApiJsonProxy)
at object lambda_method(Closure, object[])
at object Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
at void Microsoft.Tri.Infrastructure.ModuleManager.AddModules(Type[] moduleTypes)
at new Microsoft.Tri.Sensor.SensorModuleManager()
at ModuleManager Microsoft.Tri.Sensor.SensorService.CreateModuleManager()
at async Task Microsoft.Tri.Infrastructure.Service.OnStartAsync()
at void Microsoft.Tri.Infrastructure.TaskExtension.Await(Task task)
at void Microsoft.Tri.Infrastructure.Service.OnStart(string[] args)
2022-12-16 14:54:44.4234 Warn VirtualizationTypeHelper GetBiosSystemInfo Indexes:[ManufacturerIndex:1,ProductNameIndex:2,SerialNumberIndex:3,VersionIndex:0] Values:[HPE,ProLiant DL20 Gen10]
2022-12-16 14:54:45.4323 Warn Machine VirtualizationType failed [exception=System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__58.MoveNext()

Microsoft.Tri.Sensor.Updater

Spoiler

2022-12-16 14:58:41.2644 Warn GroupManagedServiceAccountImpersonationHelper GetGroupManagedServiceAccountAccessTokenAsync started [AccountName=gmsa DomainDnsName=domain1]
2022-12-16 14:58:41.2644 Warn GroupManagedServiceAccountImpersonationHelper TryGetGroupManagedServiceAccountImpersonator failed GMSA password could not be retrieved [errorCode=NoLogonServers accountName=gmsa domainDnsName=domain1]
2022-12-16 14:58:41.2644 Warn GroupManagedServiceAccountImpersonationHelper GetGroupManagedServiceAccountAccessTokenAsync started [AccountName=gmsa DomainDnsName=domain2]
2022-12-16 14:58:41.2644 Warn GroupManagedServiceAccountImpersonationHelper TryGetGroupManagedServiceAccountImpersonator failed GMSA password could not be retrieved [errorCode=NoSuchDomain accountName=gmsa domainDnsName=domain2]
2022-12-16 14:58:41.2644 Warn GroupManagedServiceAccountImpersonationHelper GetGroupManagedServiceAccountAccessTokenAsync started [AccountName=gmsa DomainDnsName=domain3]
2022-12-16 14:58:41.2644 Warn GroupManagedServiceAccountImpersonationHelper TryGetGroupManagedServiceAccountImpersonator failed GMSA password could not be retrieved [errorCode=NoSuchDomain accountName=gmsa domainDnsName=domain3]

Microsoft.Tri.Sensor.Updater-Errors

Spoiler

2022-12-16 14:58:22.0326 Error ServiceControllerExtension ChangeServiceStatus failed to change service status [name=AATPSensor status=Running Exception=System.ServiceProcess.TimeoutException: Time out has expired and the operation has not been completed.
at System.ServiceProcess.ServiceController.WaitForStatus(ServiceControllerStatus desiredStatus, TimeSpan timeout)
at Microsoft.Tri.Infrastructure.ServiceControllerExtension.ChangeServiceStatus(String name, ServiceControllerStatus status, TimeSpan timeout, Nullable`1 awaitedStatus)]

Microsoft.Tri.Sensor-Errors

Spoiler

2022-12-16 15:04:31.5835 Error DirectoryServicesClient+<CreateLdapConnectionAsync>d__48 Microsoft.Tri.Infrastructure.ExtendedException: CreateLdapConnectionAsync failed [DomainControllerDnsName=EUR-NT-CTLPTdomain1]
at async Task<LdapConnection> Microsoft.Tri.Sensor.DirectoryServicesClient.CreateLdapConnectionAsync(DomainControllerConnectionData domainControllerConnectionData, bool isGlobalCatalog, bool isTraversing)
at async Task<bool> Microsoft.Tri.Sensor.DirectoryServicesClient.TryCreateLdapConnectionAsync(DomainControllerConnectionData domainControllerConnectionData, bool isGlobalCatalog, bool isTraversing)
2022-12-16 15:04:31.5985 Error DirectoryServicesClient Microsoft.Tri.Infrastructure.ExtendedException: Failed to communicate with configured domain controllers [ _domainControllerConnectionDatas=EUR-NT-CTLPTdomain1]
at new Microsoft.Tri.Sensor.DirectoryServicesClient(IConfigurationManager configurationManager, IDirectoryServicesDomainNetworkCredentialsManager domainNetworkCredentialsManager, IDomainTrustMappingManager domainTrustMappingManager, IRemoteImpersonationManager remoteImpersonationManager, IMetricManager metricManager, IWorkspaceApplicationSensorApiJsonProxy workspaceApplicationSensorApiJsonProxy)
at object lambda_method(Closure, object[])
at object Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
at void Microsoft.Tri.Infrastructure.ModuleManager.AddModules(Type[] moduleTypes)
at new Microsoft.Tri.Sensor.SensorModuleManager()
at ModuleManager Microsoft.Tri.Sensor.SensorService.CreateModuleManager()
at async Task Microsoft.Tri.Infrastructure.Service.OnStartAsync()
at void Microsoft.Tri.Infrastructure.TaskExtension.Await(Task task)
at void Microsoft.Tri.Infrastructure.Service.OnStart(string[] args)

This is what i'd tried to do and some other infos:

Connectivity test with the .sensorapi.atp.azure.com URL -> OK

Installation with admin permissions -> OK

Testing GMSA Account connectivity with Test-ADServiceAccount -Identity 'gmsa' -> OK

Testing GMSA Account connectivity with Get-ADServiceAccount gmsa -Properties PrincipalsAllowedToRetrieveManagedPassword -> OK

AATPSensor and AATPSensorUpdater set to Disabled
AATPSensor and AATPSensorUpdater stopped
gmsa account installed with Install-ADServiceAccount gmsa
AATPSensor and AATPSensorUpdater set to Auto
AATPSensorUpdater started

NIC Teaming is enabled

FYI : I successfully installed the ATP sensor on servers that share the same configuration than this one (Same forest/domain/Network/winver)

[#2]

@Hjuanola Did you check this?

https://learn.microsoft.com/en-us/defender-for-identity/troubleshooting-known-issues#sensor-failed-t...

[#3]

@Hjuanola 

Please verify the Log on as a Service setting, as described here: https://learn.microsoft.com/en-us/defender-for-identity/directory-service-accounts#verify-that-the-g...

and verify that the latest (17th) Nov. 2022 updates have been installed on the server.

More details here: 

November 2022 Out of Band update released! Take action! - Microsoft Community Hub

What happened to Kerberos Authentication after installing the November 2022/OOB updates? - Microsoft...

[#4]

I finally found what caused the problem and it's pretty dumb : A firewall rule blocked all web traffic on this particular VM so the ATP sensor could not communicate properly with Azure...
Thanks for your answers !

[#5]

@Hjuanola 

Fixed it on Server 2016 for a customer. Needed to install .NET 4.8