ATA Services suddenly failing to start.

[David McAllister]

Hi Guys,

After my DCs have recently rebooted, the ATA lightwieght gateway agent on them is failing to start.

Looking at the error log is see the following error repeatedly:

4832 4   Error [WebClient+<InvokeAsync>d__8`1] System.Net.Http.HttpRequestException: PostAsync failed [requestTypeName=UpsertGatewayMonitoringAlertRequest] ---> System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to read data from the transport connection: The connection was closed

Anyone have any suggestions as to what might be causing this?

Thanks

David

[#2]

Looks like the DCs lost communication with the center machine.

Are you able to browse the Console UI using Internet Explorer from these machines?

[#3]

Hi Eli, I sure can. DNS seems fine as well, I can resolve the FQDN without issue.

[#4]

Which ATA version?

[#5]

Were running 1.9.7312.32791

[#6]

Are all the GWs failing or just some?

Are there any recent/ repetitive errors in the  Center's textual logs?

[#7]

4 of the 11 are failing. It might be that only these 4 have rebooted recently.

Looking at the Centre logs I i see this error repeatedly:

4496 96 Error [ExceptionFilterStream] System.IO.IOException ---> System.Net.HttpListenerException: The I/O operation has been aborted because of either a thread exit or an application request
at System.Net.HttpRequestStream.Read(Byte[] buffer, Int32 offset, Int32 size)

[#8]

At this point I would suggest to open a ticket with support where they can look more closely on the full logs, and give specific instruction for more data collection.

What you can do to get more data, is for one of the DC's that is still working fine,  do not reboot the DC itself, but restart the GW service only, and see if it can start or fails like the others.

[#9]

Thanks Eli, I guess Ill just have to open a case. I was able to successfully restart the gateway service on a machine not affected by this issue.

[#10]

Quick update for anyone experiencing the same issue. In our instance the connection was being dropped by our Tipping Point IPS.

Once we whitelisted the detection the gateways reconnected immediately.

Heres the description of the OpenSSL vulnerability it was detecting between the gateways and the ATA Centre. Im still engaged with MS to see if this can be resolved rather than just whitelisted.

Description
This filter detects an attempt to exploit a security bypass vulnerability in OpenSSL.

The specific flaw exists within how ChangeCipherSpec messages are handled by the client. An attacker can leverage this vulnerability to decrypt and inject traffic resulting in affecting the security policy of the current process.

User authentication in not required to exploit this vulnerability.

References:

Common Vulnerabilities and Exposures
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-0224

Vendor Advisory
http://www.openssl.org/news/secadv_20140605.txt

SecurityFocus Bugtraq ID
http://www.securityfocus.com/bid/67899