Experiencing Webtest activation Notificaiton failure issue in Azure Portal - 06/06 - Resolved
Published Jun 06 2019 10:51 AM 2,193 Views
Final Update: Thursday, 06 June 2019 18:41 UTC

We've confirmed that all systems are back to normal with no customer impact as of June 6 2019, 18:32 UTC. Our logs show the incident started on June 6,2019 08:05 UTC and during impact timeline that it took to resolve the issue 100% of customers who subscribed to Webtest Alert Activation notification would have missed receiving their activation notification emails.
  • Root Cause: The failure was due to downstream notification service.
  • Lessons Learned: We are looking to the telemetry for improvements on our ability to detect such issues sooner.
  • Incident Timeline: 8 Hours & 27 minutes - June 6,2019 08:05 UTC through June 6 2019, 18:32 UTC

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.


-chandar

Update: Thursday, 06 June 2019 17:42 UTC

Root cause has been isolated to a template in one of the downstream notification service which was impacting Webtest alert mail activated notifications. We are deploying a hotfix to address this issue .  Gen2 alerts are working as expected.  We estimate 2 hours before all impacts are addressed.
  • Work Around: None
  • Next Update: Before 06/06 20:00 UTC
-chandar

Version history
Last update:
‎Jun 06 2019 11:49 AM
Updated by: