Experiencing Alerting Failure issue in Azure Portal for Many Data Types - 06/14 - Resolved

Published Jun 14 2022 06:23 AM 544 Views
Final Update: Tuesday, 14 June 2022 15:50 UTC

We've confirmed that all systems are back to normal with no customer impact as of 06/14, 15:50 UTC. Our logs show the incident started on 06/14, 12:03 UTC and that during the 3 hours & 47 minutes that it took to resolve the issue customers might have experienced not receiving the alert notification, and Alerts fired during this time won't show in the portal for all the regions in Mooncake and Fairfax.
  • Root Cause: The failure was due the backend dependencies become unhealthy.
  • Incident Timeline: 3 Hours & 47 minutes - 06/14, 12:03 UTC through 06/14, 15:50 UTC.
We understand that customers rely on Azure monitor/ Azure alerts  as a critical service and apologize for any impact this incident caused.

-Srikanth

Update: Tuesday, 14 June 2022 13:07 UTC

We are aware of issues within Azure alerts service and are actively investigating. Some customers may experience not receiving the alert notification, and Alerts fired during this time won't show in the portal for all the regions in Mooncake and Fairfax. initial findings indicate that the problem began at 06/13 23:58 UTC for Fairfax and 06/14 12:03 UTC for Moon cake region. We currently have no estimate for resolution.
  • Work Around: None.
  • Next Update: Before 06/14 19:30 UTC
-Srikanth

%3CLINGO-SUB%20id%3D%22lingo-sub-3504912%22%20slang%3D%22en-US%22%3EExperiencing%20Alerting%20Failure%20issue%20in%20Azure%20Portal%20for%20Many%20Data%20Types%20-%2006%2F14%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-3504912%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Tuesday%2C%2014%20June%202022%2015%3A50%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2006%2F14%2C%2015%3A50%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2006%2F14%2C%2012%3A03%20UTC%20and%20that%20during%20the%203%20hours%20%26amp%3B%2047%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%20customers%20might%20have%20experienced%20not%20receiving%20the%20alert%20notification%2C%20and%20Alerts%20fired%20during%20this%20time%20won't%20show%20in%20the%20portal%20for%20all%20the%20regions%20in%20Mooncake%20and%20Fairfax.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20the%20backend%20dependencies%20become%20unhealthy.%3C%2FLI%3E%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%203%20Hours%20%26amp%3B%2047%20minutes%20-%2006%2F14%2C%2012%3A03%20UTC%20through%2006%2F14%2C%2015%3A50%20UTC.%3C%2FLI%3E%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Azure%20monitor%2F%20Azure%20alerts%26nbsp%3B%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Srikanth%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Tuesday%2C%2014%20June%202022%2013%3A07%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Azure%20alerts%20service%20and%20are%20actively%20investigating.%20Some%20customers%20may%20experience%20not%20receiving%20the%20alert%20notification%2C%20and%20Alerts%20fired%20during%20this%20time%20won't%20show%20in%20the%20portal%20for%20all%20the%20regions%20in%20Mooncake%20and%20Fairfax.%20initial%20findings%20indicate%20that%20the%20problem%20began%20at%2006%2F13%2023%3A58%20UTC%20for%20Fairfax%20and%2006%2F14%2012%3A03%20UTC%20for%20Moon%20cake%20region.%20We%20currently%20have%20no%20estimate%20for%20resolution.%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None.%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2006%2F14%2019%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Srikanth%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E
Version history
Last update:
‎Jun 14 2022 09:07 PM
Updated by: