Final Update: Friday, 11 February 2022 03:34 UTC
We've confirmed that all systems are back to normal with no customer impact as of 02/11, 02:35 UTC. Our logs show the incident started on 02/10, 17:30 UTC and that during the 9 hours that it took to resolve the issue approximately 80,000 customers in the East US 2 region experienced delayed data ingestion and misfired or delayed log search alerts.
- Root Cause: The failure was due to failed backend system.
- Incident Timeline: 9 Hours & 5 minutes - 02/10, 17:30 UTC through 02/11, 02:35 UTC
We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused.
-Jack Cantwell
Update: Friday, 11 February 2022 00:52 UTC
Root cause has been isolated to an outage in a backend system which caused ingested data to back up and not be processed. To address this issue, the team responsible for the backend system addressed the issue. Azure Log Analytics is now working as expected, though ingestion is still backed up and data sent to the system since the beginning of the incident is still processing. Therefore, some customers will experience continued ingestion latency or missed or latent Log Search Alerts. We estimate 3 hours before all latency and alert issues are fully resolved.
- Work Around: None
- Next Update: Before 02/11 04:00 UTC
-Jack Cantwell
Update: Thursday, 10 February 2022 23:10 UTC
Correction from previous posts: issue is in East US 2, not East US. Next update in 2 hours.
We continue to investigate issues within Log Analytics. Root cause is not fully understood at this time. Some customers continue to experience delayed or missed Log Search Alerts and latency in ingesting data. We are working to establish the start time for the issue, initial findings indicate that the problem began at 02/10 ~17:30 UTC. We currently have no estimate for resolution.
- Work Around: none
- Next Update: Before 02/10 23:00 UTC
-Jack Cantwell
Update: Thursday, 10 February 2022 23:04 UTC
We continue to investigate issues within Log Analytics. Root cause is not fully understood at this time. Some customers continue to experience delayed or missed Log Search Alerts and latency in ingesting data. We are working to establish the start time for the issue, initial findings indicate that the problem began at 02/10 ~17:30 UTC. We currently have no estimate for resolution.
- Work Around: none
- Next Update: Before 02/10 23:00 UTC
-Jack Cantwell
Update: Thursday, 10 February 2022 21:41 UTC
We continue to investigate issues within Log Analytics. Root cause is not fully understood at this time. Some customers continue to experience delayed or missed Log Search Alerts and latency in ingesting data. We are working to establish the start time for the issue, initial findings indicate that the problem began at 02/10 ~17:30 UTC. We currently have no estimate for resolution.
- Work Around: none
- Next Update: Before 02/10 23:00 UTC
-Jack Cantwell
Initial Update: Thursday, 10 February 2022 20:30 UTC
We are aware of issues within Log Analytics and are actively investigating. Some customers may experience delayed or missed Log Search Alerts and latency in ingesting data to be available for queries.
- Work Around: none
- Next Update: Before 02/10 21:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Jack Cantwell
