Final Update: Wednesday, 27 November 2019 16:44 UTC
We've confirmed that all systems are back to normal with no customer impact as of 11/27, 15:45 UTC. Our logs show the incident started on 11/19, 21:00 UTC and that during the 7 Days, 18 hours and 45 minutes that it took to resolve the issue customers would have seen error ID 6000 in their event log with a source of Service Connector or error 1209 with a source of HealthService.
- Root Cause: The failure was due to SHA2 authentication configuration issue.
- Incident Timeline: 7 Days, 18 Hours & 45 minutes - 11/19, 21:00 UTC through 11/27, 15:45 UTC
We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused.
-Rama
Update: Wednesday, 27 November 2019 11:17 UTC
Root cause has been isolated to an SHA2 authentication configuration issue impacting agent intelligence packs and management packs on Windows 7 and Windows Server 2008R2. Impacted customers would see error ID 6000 in their event log with a source of Service Connector or error 1209 with a source of HealthService. We have re-signed the patch with the SHA-1 Hashing algorithm which is supported by Windows 7 or Windows Server 2008 and earlier. We are updating the management pack configuration to mitigate the problem without customer action.
- Work Around: None
- Next Update: Before 11/27 17:30 UTC
-Rama
Update: Wednesday, 27 November 2019 06:27 UTC
We are still working hard to deploy the solution. There is currently no work around for this issue. We apologize for any inconvenience.
- Next Update: Before 11/27 10:30 UTC
-Ian
Update: Wednesday, 27 November 2019 04:43 UTC
We are currently deploying updates to mitigate the issue.
- Next Update: Before 11/27 07:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Ian
Update: Tuesday, 26 November 2019 21:53 UTC
Root cause has been isolated to an SHA2 authentication configuration issue impacting agent intelligence packs and management packs on Windows 7 and Windows Server 2008R2. Impacted customers would see error ID
6000 in their event log with a source of
Service Connector or error
1209 with a source of
HealthService. We are currently working on updating the management pack configuration to mitigate the problem without customer action. However there is a workaround that can be applied if needed.
-Jeff
Initial Update: Tuesday, 26 November 2019 18:10 UTC
We are aware of issues within Log Analytics and are actively investigating. Some customers running Windows Server 2008 R2 may see errors in event logs and clients agents may not be able to communicate back to the service properly.- Work Around: none
- Next Update: Before 11/26 20:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Ian
