Final Update: Monday, 25 November 2019 10:01 UTC
We've confirmed that all systems are back to normal with no customer impact as of 11/25, 02:30 UTC. Our logs show the incident started on 11/25, 01:10 UTC and that during the 1 hour & 20 minutes that it took to resolve the issue some customers may not receive expected alerts or
experience increased latency in receiving these alerts in West Europe region.
-
Root Cause: The failure was due to issue with one of our backend service which became unhealthy.
- Incident Timeline: 1 Hour & 20 minutes - 11/25, 01:10 UTC through 11/25, 02:30 UTC
We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused.
-satya
Update: Monday, 25 November 2019 06:45 UTC
Root cause has been isolated to latency in processing messages from storage, customers might have experienced latency or failure in receiving alerts in West Europe region. To address this issue backend service resources are been scaled out.
- Work Around: None
- Next Update: Before 11/25 11:00 UTC
-satya
Initial Update: Monday, 25 November 2019 02:28 UTC
We are aware of issues within Log Search Alerts and are actively investigating. Customers may not receive expected alerts or may experience increased latency in receiving these alerts in WEU region.
- Work Around: None
- Next Update: Before 11/25 06:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Subhash
