ATP sensor fails to start since yesterday

[ingo-boettcher]

Hi there,

we run the ATP sensor with a gMSA account on all domain controllers. Yesterday we restarted all machines because of January patch day and now the ATP sensor will get stuck while starting.

Funny: there are more than 40 DC's. The service is still starting on exactly one (!) DC. It can be restarted on this DC without any issues. All others show this error.

Rebooting the machines will not help.

2024-01-24 16:24:50.9788 Info RemoteImpersonationManager CreateImpersonatorInternalAsync started [UserName=mdiuser$ Domain=domain.local IsGroupManagedServiceAccount=True]
2024-01-24 16:24:51.4632 Info RemoteImpersonationManager GetGroupManagedServiceAccountTokenAsync finished [UserName=mdiuser$ Domain=domain.local IsSuccess=False]
2024-01-24 16:24:51.4632 Info RemoteImpersonationManager CreateImpersonatorInternalAsync finished [UserName=mdiuser$ Domain=domain.local]
2024-01-24 16:24:51.4632 Warn DirectoryServicesClient CreateLdapConnectionAsync failed to retrieve group managed service account password. [DomainControllerDnsName=dc03.domain.local Domain=domain.local UserName=mdiuser$ ]

We have not changed anything regarding sensors or the gMSA account for months, so this configuration was running without issues until yesterday.

Running Test-ADServiceAccount -Identity "mdiuser" on the affected machines gives "True", so the machine can successfully retrieve the gMSA password. 

I have checked that the mdiuser account is part of the GPO that allows logon as service on all machines. 

Now I am running out of ideas. The system tells me, it can access the gMSA password, the agent tells me it can't. Whats wrong?

Best regards, Ingo

[#2]

Were you able to get this resolved? I have the same issue since updating to release 2.227. I created a new gMSA account, and got the sensors started using the new gMSA.

They ran fine for a couple of weeks until they update to release 2.228. Now I am seeing "failed to retrieve group managed service account password" for the new gMSA

[#3]

Unfortunately not yet.

I have a support ticket open for two weeks now. The Microsoft support has asked a few things but there has been no solution yet.

[#4]

Have you had any luck? We have an open support case as well but it's taking a long time.

[#5]

No. I had a long support case with MS and we did not find a real solution. At the end they asked me to create yet another gMSA and it was working then for a few days. Exactly long enough to close the case.

Now a few days later some of the DC's show the same error again. I think, I'll go back to a normal account. Seems like this gMSA stuff is somehow broken.

[#6]

• @ingo-boettcher does your kds-rootkey possibly point to a demoted/deleted DC? 

[#7]

there were two root keys, where one pointed to an old DC that was no longer in use for years. I have removed this old key and will see what happens.

But the key was from 2013, the old DC removed at least five years ago and the gMSA was working with ATP for some years... but we'll see!

[#8]

Please let me know

[#9]

I'm getting the same error (version 2.239.18125.50420). KDS root key was pointing to a demoted DC, but reassociating it to a current DC has made no difference.
Similar scenario to the OP, where one DC out of 20+ is fine, but the others are busted. Strangely, the GSMA is being used across a forest trust and those DCs are all fine.

[#10]

@M___T  

So in the end, our issue turned out to be in relation to the awful Secure Time Seeding service, which caused a domain controller to jump into the future by two months and back again. During that interval, it seems that our GMSA did a password reset, and now the "lastlogondate" and "passwordlastset" dates are still in the future, so now none of the DCs (other than the one where the time jump occurred) can retrieve the password.
These are the selected date properties for the account - note the password/lastlogon dates are still a month from now.

Get-ADServiceAccount -Identity [SVC_ACCOUNT] -Properties *
...
badPasswordTime                            : 133693679012898018
badPwdCount                                : 28
LastBadPasswordAttempt                     : 29/08/2024 11:18:21 AM
lastLogon                                  : 133690172347063244
LastLogonDate                              : 17/09/2024 1:27:15 PM
lastLogonTimestamp                         : 133710172352439465
PasswordLastSet                            : 17/09/2024 1:27:15 PM
pwdLastSet                                 : 133710172352439465

In a domain environment, where your DCs are syncing with a good NTP source, you MUST disable this STS garbage, which is enabled by default. Windows feature that resets system clocks based on random data is wreaking havoc | Ars Technica

The only fix for us now, according to Microsoft Support, is to create a new GMSA ...or wait another month until it starts working.

[#11]

Thanks for this info! Troubleshooting the same thing and with your help discovered 2 out of 5 gmsa accounts have the same issue. If we recreate the accounts what is to stop it from happening again? Any info or articles is greatly appreciated!

[#12]

To disable the secure time seeding feature, go to the below mentions registry key and set the registry value to ‘0’ for the following Registry Key (or GPO of course 😉 😞
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config
Value Name: UtilizeSslTimeData
Value Type: REG_DWORD