Experiencing alerting failures for many mail notifications - 04/18 - Resolved

[Azure-Monitor-Team]

Final Update: Thursday, 18 April 2019 21:25 UTC

We've confirmed that all systems are back to normal with no customer impact as of 4/18, 20:25 UTC.

• Root Cause: The failure was due to a dependent service used for email communication which entered an unhealthy state.
• Lessons Learned: We will be following up to understand how this issue could have been detected and mitigated more quickly.

We understand that customers rely on Azure Monitor services and apologize for any impact this incident caused.

-Matthew Cosner

---

Update: Thursday, 18 April 2019 20:14 UTC

Root cause has been isolated to a dependent service which was in an unhealthy state.  This service is in the process of rolling back to an earlier build.  Some customers may continue to experience alerting delays or failures.

• Work Around:  Email notifications configured to send directly to an email address are not impacted.
• Next Update:  Before 04/18 22:30 UTC

-Matthew Cosner

---

Initial Update: Thursday, 18 April 2019 18:54 UTC

We are aware of issues with email notifications either failing to be sent or being delayed.  This impact includes some subset of customers using classic alert emails, daily cap email notifications, smart detection email notifications, and availability test email notifications configured to send to subscription admins, contributors, or readers.  Other notification flows such as webhooks are not impacted.  Informational notifications were also sent to affected subscriptions through the Azure portal.

• Work Around: Email notifications configured to send directly to an email address are not impacted.
• Next Update: Before 04/18 20:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Matthew Cosner

---