Experiencing Data Latency in Azure portal for Activity Log Alerts - 06/04 - Resolved
Published Jun 04 2020 06:22 AM 1,374 Views
Final Update: Thursday, 04 June 2020 19:20 UTC

We've confirmed that all systems are back to normal with no customer impact as of 6/4, 18:54 UTC. Our logs show the incident started on 6/4, 12:05 UTC and that during the 6 hours and 49 minutes that it took to resolve the issue customers experienced delayed log search alerts by up to 2.5 hours.
  • Root Cause: The failure was due to the service reaching an operating limit as a result of large unexpected spike in alert notifications across regions.
  • Incident Timeline: 6 Hours & 49 minutes - 6/4, 12:05 UTC through 6/4, 18:54 UTC
We understand that customers rely on Activity Log Alerts as a critical service and apologize for any impact this incident caused.

-Jeff

Update: Thursday, 04 June 2020 17:06 UTC

Root cause has been isolated to an extremely large spike in log search alerts across multiple regions which was impacting activity log alerts ability to process alerts in a timely manner. To address this issue we have scaled out the service and the primary incident causing the spike has been mitigated. Some customers may continue experience delayed log search alerts of up to 2 hours. We estimate 2-3 hours before timely log search alerts are being processed again.
  • Work Around: None
  • Next Update: Before 06/04 21:30 UTC
-Jeff

Initial Update: Thursday, 04 June 2020 13:18 UTC

We are aware of issues within Activity Log Alerts and are actively investigating. Some customers may experience delayed alerts. Alerts would have eventually fired.
    • Work Around: None
    • Next Update: Before 06/04 17:30 UTC
    We are working hard to resolve this issue and apologize for any inconvenience.
    -Anmol

    Version history
    Last update:
    ‎Jun 04 2020 12:32 PM
    Updated by: