Azure Advanced Threat Protection service failing to start after installing November MS Patches

[BC-ITGuy]

Azure Advanced Threat Protection service failing to start after installing November MS Patches:

** .NET CU - KB5020627 - November 8, 2022-KB5020627 Cumulative Update for .NET Framework 3.5 and 4.7.2 for Windows 10, version 1809 and Windows Server 2019
** WINDOWS 2019 OS CU - KB5019966 - Cumulative Update for Windows Server 2019 for x64-based Systems (KB5019966)

Here's the error from the "C:\Program Files\Azure Advanced Threat Protection Sensor\2.193.15824.20477\Logs\Microsoft.Tri.Sensor.log" log:

022-11-16 07:09:38.0502 Info RemoteImpersonationManager CreateImpersonatorInternalAsync started [UserName=%OURGMSA% Domain=%SOMEDOMAIN% IsGroupManagedServiceAccount=True]
2022-11-16 07:09:38.0658 Info RemoteImpersonationManager GetGroupManagedServiceAccountTokenAsync finished [UserName=%OURGMSA% Domain=%SOMEDOMAIN% IsSuccess=False]
2022-11-16 07:09:38.0658 Info RemoteImpersonationManager CreateImpersonatorInternalAsync finished [UserName=%OURGMSA% Domain=%SOMEDOMAIN%]
2022-11-16 07:09:38.0814 Warn DirectoryServicesClient CreateLdapConnectionAsync failed to retrieve group managed service account password. [DomainControllerDnsName=%DOMAINCONTROLLER01%.%SOMEDOMAIN% Domain=%SOMEDOMAIN% UserName=%OURGMSA% ]
2022-11-16 07:09:38.0970 Info DirectoryServicesClient CreateLdapConnectionAsync failed to connect [DomainControllerDnsName=%DOMAINCONTROLLER01%.%SOMEDOMAIN% Domain=s%SOMEOTHERDOMAIN% UserName=%SOMESERVICEACCOUNT% ResultCode=82]
2022-11-16 07:09:38.0970 Info DirectoryServicesClient CreateLdapConnectionAsync failed to connect [DomainControllerDnsName=%DOMAINCONTROLLER01%.%SOMEDOMAIN% Domain=%ANOTHERDOMAIN% UserName=%SOMESERVICEACCOUNT% ResultCode=82]
2022-11-16 07:09:38.1126 Info DirectoryServicesClient CreateLdapConnectionAsync failed to connect [DomainControllerDnsName=%DOMAINCONTROLLER01%.%SOMEDOMAIN% Domain=%YETANOTHERDOMAIN% UserName=%SOMESERVICEACCOUNT% ResultCode=82]
2022-11-16 07:09:38.1126 Info RemoteImpersonationManager CreateImpersonatorInternalAsync started [UserName=%SOMEOTHERGMSA% Domain=%SOMEOTHERDOMAIN% IsGroupManagedServiceAccount=True]
2022-11-16 07:09:53.1035 Info RemoteImpersonationManager GetGroupManagedServiceAccountTokenAsync finished [UserName=%SOMEOTHERGMSA% Domain=%SOMEOTHERDOMAIN% IsSuccess=False]
2022-11-16 07:09:53.1035 Info RemoteImpersonationManager CreateImpersonatorInternalAsync finished [UserName=%SOMEOTHERGMSA% Domain=%SOMEOTHERDOMAIN%]
2022-11-16 07:09:53.1035 Warn DirectoryServicesClient CreateLdapConnectionAsync failed to retrieve group managed service account password. [DomainControllerDnsName=%DOMAINCONTROLLER01%.%SOMEDOMAIN% Domain=%SOMEOTHERDOMAIN% UserName=%SOMEOTHERGMSA% ]
2022-11-16 07:09:53.1035 Info DirectoryServicesClient CreateLdapConnectionAsync failed to connect [DomainControllerDnsName=%DOMAINCONTROLLER01%.%SOMEDOMAIN% Domain=%YETYETANOTHERDOMAIN% UserName=svc-prod-azureatp ResultCode=82]
2022-11-16 07:09:53.1035 Info DirectoryServicesClient CreateLdapConnectionAsync failed to connect [DomainControllerDnsName=%DOMAINCONTROLLER01%.%SOMEDOMAIN% Domain=%YETYETYETANOTHERDOMAIN%UserName=svc-ATP ResultCode=82]
2022-11-16 07:09:53.1192 Info DirectoryServicesClient CreateLdapConnectionAsync failed to connect [DomainControllerDnsName=%DOMAINCONTROLLER01%.%SOMEDOMAIN% Domain=%YETYETYETYETANOTHERDOMAIN% UserName=%ANOTHERSOMESERVICEACCOUNT% ResultCode=82]
2022-11-16 07:09:53.1192 Info DirectoryServicesClient CreateLdapConnectionAsync failed to connect [DomainControllerDnsName=%DOMAINCONTROLLER01%.%SOMEDOMAIN% Domain=%YETYETYETYETYETANOTHERDOMAIN% UserName=%SOMESERVICEACCOUNT% ResultCode=82]
2022-11-16 07:09:53.1192 Info DirectoryServicesClient CreateLdapConnectionAsync failed to connect [DomainControllerDnsName=%DOMAINCONTROLLER01%.%SOMEDOMAIN% Domain=%YETYETYETYETYETANOTHERDOMAIN% UserName=%SOMESERVICEACCOUNT% ResultCode=82]
2022-11-16 07:09:53.1348 Info DirectoryServicesClient CreateLdapConnectionAsync failed to connect [DomainControllerDnsName=%DOMAINCONTROLLER01%.%SOMEDOMAIN% Domain=%YETYETYETYETYETYETANOTHERDOMAIN%UserName=%SOMESERVICEACCOUNT% ResultCode=82]
2022-11-16 07:09:54.2441 Error DirectoryServicesClient+<CreateLdapConnectionAsync>d__47 Microsoft.Tri.Infrastructure.ExtendedException: CreateLdapConnectionAsync failed [DomainControllerDnsName=%DOMAINCONTROLLER01%.%SOMEDOMAIN%]
at async Task<LdapConnection> Microsoft.Tri.Sensor.DirectoryServicesClient.CreateLdapConnectionAsync(DomainControllerConnectionData domainControllerConnectionData, bool isGlobalCatalog, bool isTraversing)
at async Task<bool> Microsoft.Tri.Sensor.DirectoryServicesClient.TryCreateLdapConnectionAsync(DomainControllerConnectionData domainControllerConnectionData, bool isGlobalCatalog, bool isTraversing)
2022-11-16 07:09:54.2754 Error DirectoryServicesClient Microsoft.Tri.Infrastructure.ExtendedException: Failed to communicate with configured domain controllers [ _domainControllerConnectionDatas=%DOMAINCONTROLLER01%.%SOMEDOMAIN%]
at new Microsoft.Tri.Sensor.DirectoryServicesClient(IConfigurationManager configurationManager, IDirectoryServicesDomainNetworkCredentialsManager domainNetworkCredentialsManager, IDomainTrustMappingManager domainTrustMappingManager, IRemoteImpersonationManager remoteImpersonationManager, IMetricManager metricManager, IWorkspaceApplicationSensorApiJsonProxy workspaceApplicationSensorApiJsonProxy)
at object lambda_method(Closure, object[])
at object Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
at void Microsoft.Tri.Infrastructure.ModuleManager.AddModules(Type[] moduleTypes)
at new Microsoft.Tri.Sensor.SensorModuleManager()
at ModuleManager Microsoft.Tri.Sensor.SensorService.CreateModuleManager()
at async Task Microsoft.Tri.Infrastructure.Service.OnStartAsync()
at void Microsoft.Tri.Infrastructure.TaskExtension.Await(Task task)

FYI:

Following this document, we confirmed that the gMSA is setup correctly.

https://learn.microsoft.com/en-us/defender-for-identity/troubleshooting-known-issues#sensor-failed-t...

Again, this all started after patching and rebooting.

Is this a known issue with the Nov patches?

Any assistance on this is greatly appreciated!

[#2]

@BC-ITGuy 

We've seen several cases where the sensors failed because of the November updates and how they affect Kerberos and gMSAs in particular.

There's some information in the links below (please note that those are not Microsoft official docs), but the best approach would be for you to open a support ticket and get the best solution for your environment.

Spend some Time on Properly Configuring and Monitoring your Domain Controllers this Patch Tuesday - ...

Windows Kerberos authentication breaks after November updates (bleepingcomputer.com)

[#3]

@Martin_Schvartzman 

Thanks for the helpful info.

I confirmed we're receiving the event id 14 for the atp group managed service account as outlined here:

https://www.bleepingcomputer.com/news/microsoft/windows-kerberos-authentication-breaks-after-novembe...

event id 14:

While processing an AS request for target service krbtgt, the account %OURGMSAUSEDWITHATP%$ did not have a suitable key for generating a Kerberos ticket (the missing key has an ID of 1). The requested etypes : 18 17 23 24 -135 3. The accounts available etypes : 23 18 17. Changing or resetting the password of %OURGMSAUSEDWITHATP%$ will generate a proper key.

--

It doesn't look like there's a way to force a password change on a gmsa to see if this message is actually accurate. The time change is hardcoded at creation (30 days) (https://social.technet.microsoft.com/Forums/en-US/d08bdb51-81f4-4368-9213-33a969e1b29b/powershell-cm...)

Maybe we can create a new gmsa and test it with atp? Would this work or is the message not accurate?

It says that Microsoft is working on a fix for this issue and it will be released in the next few weeks.

[#4]

Microsoft released a updated CU to resolve this issue: https://www.bleepingcomputer.com/news/microsoft/microsoft-fixes-windows-kerberos-auth-issues-in-emer...