Final Update: Thursday, 20 September 2018 09:58 UTC
We've confirmed that all systems are back to normal with no customer impact as of 09/20, 08:35 UTC. Our logs show the incident started on 09/19, 11:00 UTC and that during the 21 hours 35 minutes that it took to resolve the customers would have seen failure while accessing Azure Log Analytics services in Azure & OMS portal.
-
Root Cause: A backend Log Analytics service experienced a large number of requests which caused failures in several other dependent services.
-
Incident Timeline: 21 Hours & 35 minutes - 09/19, 11:00 UTC through 09/20, 08:35 UTC
We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused.
-Mohini
Update: Thursday, 20 September 2018 06:13 UTC
We continue to investigate issues within Log Analytics. Customers in East US region continue to experience difficulties accessing data, high latency and timeouts when getting workspace information, running Log Analytics queries, and other operations related to Log Analytics workspaces. Also some customers may see issue onboarding new agents. We identify the source of issue in Azure Log Analytics and are working on mitigation approach, but at present have no defined ETA for resolution.
-
Work Around: None
-
Next Update: Before 09/20 08:30 UTC
-Mohini
Final Update: Wednesday, 19 September 2018 19:35 UTC
We confirm that all our systems are back to normal as of 09/19, 18:45 UTC, however from 09/19 11:00 UTC to 09/19 18:45 UTC customers would have seen failure while accessing Azure Log Analytics services in Azure & OMS portal.
-
Root Cause: A backend Log Analytics service experienced a large number of requests which caused failures in several other dependent services.
-
Incident Timeline: 7 Hours & 45 minutes - 09/19 11:00 UTC through 09/19 18:45 UTC
We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused.
-Sapna
Update: Wednesday, 19 September 2018 18:02 UTC
We are working on mitigating issues within Azure Log Analytics. Customers in East US region continue to experience data access and data latency. We have applied mitigation to Service Map services and has no impact at the present. We identify the source of issue in Azure Log Analytics and are working on mitigation approach, but at present have no defined ETA for resolution.
-
Work Around: None
-
Next Update: Before 09/19 23:00 UTC
-Sapna
Update: Wednesday, 19 September 2018 15:41 UTC
We continue to investigate issues within Log Analytics and Service Map. Customers in East US region continue to experience latency and timeout while fetching workspace information and may not able to view recent dependency map from OMS portal. We are working to establish the start time for the issue, initial findings indicate that the problem began at 09/19 ~11:00 UTC.
-
Work Around: None
-
Next Update: Before 09/19 18:00 UTC
-Mohini
Initial Update: Wednesday, 19 September 2018 13:25 UTC
We are aware of issues within Log Analytics and Service Map and are actively investigating. Some customers may experience latency and timeout while fetching workspace information and may not able to view recent dependency map from OMS portal.
-
Work Around: None
-
Next Update: Before 09/19 15:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Mohini
