MDI sensor service will not start on ADFS server

[novaksam]

I've exhausted my ability to troubleshoot why my ADFS sensor installs just will not start, so hoping someone can provide some guidance on how to get this working :)

Info:

• Windows Server 2022 Datacenter
• Public IP, no proxy
• Using gMSA
• Sensor version: 2.203.16523.48348
• Successful installation /w gMSA on DCs

Troubleshooting:

• Verified that ADFS auditing was set to verbose
• Verified that gMSA could access database
• Verified that gMSA is allowed to logon as a service under the DCs
  • Is this need on the ADFS servers as well?
• Verified that the sensor config was given a FQDN DC.
• Verified DisableRenegoONserver is set to 0
• Verified DisableRenegoONclient is set to 0
• The dns name for our sensor endpoint is resolving correctly.

Observations:

• Microsoft.Tri.Sensor.Updater is not listening on 444, but system is
• There is an ATP certificate in the machine personal store from the installation, despite the logs saying one isn't found/used.

Log entry:

Microsoft.Tri.Sensor.log

2023-05-31 17:58:00.5355 Error ExceptionHandler Microsoft.Tri.Infrastructure.ExtendedException: RestrictCpuAsync failed, exiting ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at int System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at int System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at int System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
   at int System.Net.TlsStream.EndRead(IAsyncResult asyncResult)
   at void System.Net.Connection.ReadCallback(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at Stream System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, out TransportContext context)
   at void System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at async Task<HttpResponseMessage> System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task<HttpResponseMessage> sendTask, HttpRequestMessage request, CancellationTokenSource cts, bool disposeCts)
   at async Task<TResponse> Microsoft.Tri.Common.CommunicationWebClient.SendAsync<TResponse>(byte[] requestBytes, int offset, int count)
   at async Task<TResponse> Microsoft.Tri.Common.CommunicationWebClient.SendWithRetryAsync<TResponse>(byte[] requestBytes, int offset, int count)
   at async Task Microsoft.Tri.Common.CommunicationWebClient.SendAsync(IVoidRequest request)
   at async Task Microsoft.Tri.Sensor.Common.ServiceProxy<TWebClientConfiguration>.SendAsync(IVoidRequest request)
   at async Task Microsoft.Tri.Sensor.SensorResourceManager.RestrictCpuAsync()
   --- End of inner exception stack trace --

Microsoft.Tri.Sensor-Errors

2023-05-31 17:58:00.5355 Error ExceptionHandler Microsoft.Tri.Infrastructure.ExtendedException: RestrictCpuAsync failed, exiting ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at int System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at int System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at int System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
   at int System.Net.TlsStream.EndRead(IAsyncResult asyncResult)
   at void System.Net.Connection.ReadCallback(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at Stream System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, out TransportContext context)
   at void System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at async Task<HttpResponseMessage> System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task<HttpResponseMessage> sendTask, HttpRequestMessage request, CancellationTokenSource cts, bool disposeCts)
   at async Task<TResponse> Microsoft.Tri.Common.CommunicationWebClient.SendAsync<TResponse>(byte[] requestBytes, int offset, int count)
   at async Task<TResponse> Microsoft.Tri.Common.CommunicationWebClient.SendWithRetryAsync<TResponse>(byte[] requestBytes, int offset, int count)
   at async Task Microsoft.Tri.Common.CommunicationWebClient.SendAsync(IVoidRequest request)
   at async Task Microsoft.Tri.Sensor.Common.ServiceProxy<TWebClientConfiguration>.SendAsync(IVoidRequest request)
   at async Task Microsoft.Tri.Sensor.SensorResourceManager.RestrictCpuAsync()
   --- End of inner exception stack trace ---

Microsoft.Tri.Sensor.Updater

2023-05-31 17:58:00.2690 Warn  ClientCertificateAuthenticationMiddleware+ClientCertificateAuthenticationHandler AuthenticateCoreAsync missing client certificate
2023-05-31 17:58:00.2690 Warn  AppBuilderExtension UseExceptionHandler IOException ignored [Details=InnerExceptionType=HttpListenerException ErrorCode=1229]
2023-05-31 17:58:00.2811 Warn  ClientCertificateAuthenticationMiddleware+ClientCertificateAuthenticationHandler AuthenticateCoreAsync missing client certificate
2023-05-31 17:58:00.2811 Warn  AppBuilderExtension UseExceptionHandler IOException ignored [Details=InnerExceptionType=HttpListenerException ErrorCode=1229]
2023-05-31 17:58:00.3003 Warn  ClientCertificateAuthenticationMiddleware+ClientCertificateAuthenticationHandler AuthenticateCoreAsync missing client certificate
2023-05-31 17:58:00.3003 Warn  AppBuilderExtension UseExceptionHandler IOException ignored [Details=InnerExceptionType=HttpListenerException ErrorCode=1229]
2023-05-31 17:58:00.3316 Warn  ClientCertificateAuthenticationMiddleware+ClientCertificateAuthenticationHandler AuthenticateCoreAsync missing client certificate
2023-05-31 17:58:00.3316 Warn  AppBuilderExtension UseExceptionHandler IOException ignored [Details=InnerExceptionType=HttpListenerException ErrorCode=1229]
2023-05-31 17:58:15.1918 Warn  ClientCertificateAuthenticationMiddleware+ClientCertificateAuthenticationHandler AuthenticateCoreAsync missing client certificate

[#2]

Check acls on the cert. Make sure both local system and local service has read access to the cert.

[#3]

@Eli OfekThe ATP cert (Azure ATP Sensor) had the following:

• System
• Administrators
• AATPSensor

I added:

• gMSA
• LOCAL SERVICE

Issue persists, no changes in the logging output that I can observe.

[#4]

OK, I took a deeper look in the code that outputs this message.
The problem is that the sensor is sending a TLS request to the updater on localhost TCP 444.
It authenticates using a client certificate in the request,
But when the updater gets the request and tries to authenticate via the certificate, it is missing from the request.

So either something is blocking the sensor from putting it there, or we have some kind of "Man in the Middle" in the machine that "scrubs" the certificate out.

Are you familiar with anything installed on this machine that might temper with this request?

Some things I found related to similar cases from the past:
Having ADFS proxy installed on the same machine might cause it.

Some customers reported the issue fixed after setting those registry values to 0:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL]
"DisableRenegoOnServer"=dword:00000001
"DisableRenegoOnClient"=dword:00000001

If all fails, the next step is openings a support request, but I have to be honest with you:
The last case that did not resolved by all of the above required so much research time to check what got broken in the machine that eventually it would have been easier to just rebuild it.

[#5]

Nope nothing tampering with certs. Is the service supposed to be the process listening on 444, or should it be SYSTEM?

TCP 127.0.0.1:444 0.0.0.0:0 LISTENING 4

[#6]

Nevermind, I stopped the updater service and the listening stopped.

[#7 · Accepted solution]

The rep from Microsoft was able to resolve the startup issue!

If a server has HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\SendTrustedIssuerList set to 1, that will cause the service to fail.

According to https://learn.microsoft.com/en-us/windows-server/security/tls/what-s-new-in-tls-ssl-schannel-ssp-ove... 0 is the default as of 2012R2

Since we did in-place upgrades, it makes sense that the value would be set from those previous OS's.