AATP sensors stopped communicating

%3CLINGO-SUB%20id%3D%22lingo-sub-1570580%22%20slang%3D%22en-US%22%3EAATP%20sensors%20stopped%20communicating%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1570580%22%20slang%3D%22en-US%22%3E%3CP%3EHi%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Esince%20today%20morning%20(7%3A30%20AM%20CET)%20all%20sensors%20showing%20the%20Disconnected%20status%20message%20on%20the%20AATP%20portal.%20Checked%20the%20logs%2C%20it%20shows%20the%20following%20error%20message%3A%3CBR%20%2F%3E%3CBR%20%2F%3E%3CEM%3EError%20HttpResponseMessageExtension%20Microsoft.Tri.Infrastructure.ExtendedHttpRequestException%3A%20Response%20status%20code%20does%20not%20indicate%20success%3A%20500%20(Internal%20Server%20Error).%20---%26gt%3B%20System.Net.Http.HttpRequestException%3A%20Response%20status%20code%20does%20not%20indicate%20success%3A%20500%20(Internal%20Server%20Error).%3C%2FEM%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAs%20far%20as%20I%20know%2C%20nobody%20changed%20any%20settings%20related%20to%20the%20firewalls%20in%20our%20environment.%20We%20have%2035%20sensors%20(US-East%2C%20US-West%2C%20EMEA%2C%20APAC)%20in%20two%20domains%20and%20all%20stopped%20working%20in%20he%20same%20time.%20Around%2010%3A00%20AM%20CET%20I%20tried%20to%20login%20to%20the%20AATP%20portal%2C%20but%20the%20site%20was%20not%20reachable%2C%20that%20issue%20is%20no%20longer%20persist.%20Checked%20the%20custom%20api%20URL%20in%20a%20browser%2C%20that%20shows%20the%26nbsp%3B%26nbsp%3B%3CEM%3EHTTP%20Error%20503.%20The%20service%20is%20unavailable.%20%3C%2FEM%3Eerror%20message%2C%20about%20which%20I%20am%20not%20sure%20that%20it%20is%20expected%20or%20not%20in%20a%20browser.%20Tried%20to%20restart%20the%20sensor%20on%20a%20few%20DCs%2C%20no%20luck.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAppreciate%20any%20help!%3CBR%20%2F%3E%3CBR%20%2F%3EThanks%2C%3CBR%20%2F%3EDavid%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1570672%22%20slang%3D%22en-US%22%3ERe%3A%20AATP%20sensors%20stopped%20communicating%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1570672%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F751334%22%20target%3D%22_blank%22%3E%40dbalogh%3C%2FA%3E%26nbsp%3B%3CBR%20%2F%3EIf%20your%20workspace%20is%20in%20the%20US%20East%20region%2C%20(Check%20the%20workspace%20about%20box)%20then%20there%20was%20a%20n%20issue%20that%20was%20mitigated%20only%20a%20few%20minutes%20ago.%20Do%20yo%20still%20see%20it%20now%20%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1570687%22%20slang%3D%22en-US%22%3ERe%3A%20AATP%20sensors%20stopped%20communicating%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1570687%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F106935%22%20target%3D%22_blank%22%3E%40Eli%20Ofek%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22ng-binding%22%3EGeolocation%3CBR%20%2F%3E%3C%2FSPAN%3E%3CSPAN%20class%3D%22helpPanelAboutContentItemValue%20ng-binding%22%3ENorth%20America%20%2F%20Central%20America%20%2F%20Caribbean%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22helpPanelAboutContentItemValue%20ng-binding%22%3EStill%20seeing%20the%20error.%20Tried%20to%20restart%20the%20sensor%20on%20one%20of%20the%20DCs%2C%20but%20the%20service%20cannot%20start%20(error%201067).%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1570698%22%20slang%3D%22en-US%22%3ERe%3A%20AATP%20sensors%20stopped%20communicating%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1570698%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F751334%22%20target%3D%22_blank%22%3E%40dbalogh%3C%2FA%3E%26nbsp%3B%3CBR%20%2F%3EThis%20is%20indeed%20the%20cluster%20that%20had%20issues.%3C%2FP%3E%0A%3CP%3EAre%20you%20able%20to%20connect%20to%20the%20workspace%20portal%20without%20issues%20%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1570720%22%20slang%3D%22en-US%22%3ERe%3A%20AATP%20sensors%20stopped%20communicating%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1570720%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F106935%22%20target%3D%22_blank%22%3E%40Eli%20Ofek%3C%2FA%3E%26nbsp%3BYes%2C%20the%20connection%20to%20the%20portal%20is%20working%20fine.%20Had%20problems%20with%20it%20earlier%20this%20morning%20and%20I%20noticed%20the%20sensor%20error%20after%20I%20was%20able%20to%20log%20in.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EEdit%3A%20Seeing%20this%20error%20on%20the%20statuspage%3A%20Partial%20Connectivity%20issues%20for%20sensors%20in%20East%20US%20region%20Investigating%20-%20We%20are%20currently%20investigating%20this%20issue.%20Aug%206%2C%2014%3A13%20UTC%3C%2FP%3E%3C%2FLINGO-BODY%3E
New Contributor

Hi,

 

since today morning (7:30 AM CET) all sensors showing the Disconnected status message on the AATP portal. Checked the logs, it shows the following error message:

Error HttpResponseMessageExtension Microsoft.Tri.Infrastructure.ExtendedHttpRequestException: Response status code does not indicate success: 500 (Internal Server Error). ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 500 (Internal Server Error).

 

As far as I know, nobody changed any settings related to the firewalls in our environment. We have 35 sensors (US-East, US-West, EMEA, APAC) in two domains and all stopped working in he same time. Around 10:00 AM CET I tried to login to the AATP portal, but the site was not reachable, that issue is no longer persist. Checked the custom api URL in a browser, that shows the  HTTP Error 503. The service is unavailable. error message, about which I am not sure that it is expected or not in a browser. Tried to restart the sensor on a few DCs, no luck.

 

Appreciate any help!

Thanks,
David 

 

4 Replies
Highlighted

@dbalogh 
If your workspace is in the US East region, (Check the workspace about box) then there was a n issue that was mitigated only a few minutes ago. Do yo still see it now ?

Highlighted

@Eli Ofek 

Geolocation
North America / Central America / Caribbean

Still seeing the error. Tried to restart the sensor on one of the DCs, but the service cannot start (error 1067).

Highlighted

@dbalogh 
This is indeed the cluster that had issues.

Are you able to connect to the workspace portal without issues ?

Highlighted

@Eli Ofek Yes, the connection to the portal is working fine. Had problems with it earlier this morning and I noticed the sensor error after I was able to log in.

 

Edit: Seeing this error on the statuspage: Partial Connectivity issues for sensors in East US region Investigating - We are currently investigating this issue. Aug 6, 14:13 UTC