Experiencing Latency, Data Loss issue, and alerting issues in West Central US-03/15- Resolved

Silver Contributor

Mar 15, 2020

Final Update: Sunday, 15 March 2020 22:43 UTC

We've confirmed that all systems are back to normal with no customer impact as of 3/15, 22:25 UTC. Our logs show the incident started on 3/15, 20:14 UTC and that during the 2 hours 11 minutes that it took to resolve the issue customers experienced latent ingestion of data and log search alerts misfiring.

Root Cause: The failure was due to a utility problem at the West Central US data center.
Incident Timeline: 2 Hours & 11 minutes - 3/15, 20:14 UTC through 3/15, 22:25 UTC

We understand that customers rely on Application Insights and Azure log analytics as critical services and apologize for any impact this incident caused.

-Jeff

Update: Sunday, 15 March 2020 22:24 UTC

Root cause has been isolated to a utility issue in West Central US data center which was impacting communications. To address this issue Azure teams have resolved it. Some customers may experience data still being ingested with a small amount of latency and possible misfiring alerts until the ingestion catches up.

Next Update: Before 03/16 00:30 UTC

-Jeff

Initial Update: Sunday, 15 March 2020 21:40 UTC

We are aware of issues within Application Insights and Azure Log Analytics and are actively investigating. Some customers may experience Latency and Data Loss, configuration failures for alerts, misfired alerts, and other unexpected behaviors.