%3CLINGO-SUB%20id%3D%22lingo-sub-1111100%22%20slang%3D%22en-US%22%3EExperiencing%20issues%20viewing%20alerts%20in%20USGov%20and%20China%20region%20-%2001%2F15%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1111100%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Thursday%2C%2016%20January%202020%2008%3A09%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2001%2F16%2C%2008%3A00%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2001%2F11%2C%2004%3A25%20UTC%20and%20that%20during%20the%205%20days%203%20hours%2035%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%20some%20of%20customers%20in%20US%20Gov%20and%20Azure%20China%20Region%20might%20have%20experienced%20issue%20while%20viewing%20alerts.%3CBR%20%2F%3E%3CUL%3E%0A%20%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20an%20issue%20with%20one%20of%20our%20dependent%20service.%26nbsp%3B%3C%2FLI%3E%0A%20%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%205%20Days%2C%203%20Hours%20%26amp%3B%2035%20minutes%20-%2001%2F11%2C%2004%3A25%20UTC%20through%2001%2F16%2C%2008%3A00%20UTC%3C%2FLI%3E%0A%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Azure%20Monitor%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Monish%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Thursday%2C%2016%20January%202020%2005%3A32%20UTC%26nbsp%3B%3CBR%20%2F%3EThe%20issue%20got%20mitigated%20for%20the%20customers%20in%20US%20Gov%20Region%20as%20of%2001%2F16%2004%3A25%20UTC%20and%20the%20engineers%20are%20continuing%20to%20work%20on%20rolling%20back%20the%20service%20Azure%20China%20for%20mitigating%20the%20issue.%20Some%20customer%20in%20Azure%20China%20Region%20might%20still%20experience%20issue%20in%20viewing%20the%20alerts.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2001%2F16%2010%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Monish%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Thursday%2C%2016%20January%202020%2002%3A38%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3E%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20rgb(255%2C%20255%2C%20255)%3B%20color%3A%20rgb(0%2C%200%2C%200)%3B%20font-family%3A%20%26quot%3BHelvetica%20Neue%26quot%3B%2CHelvetica%2CArial%2Csans-serif%3B%20font-size%3A%2014px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20400%3B%20letter-spacing%3A%20normal%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3ERoot%20cause%20has%20been%20isolated%20to%20recent%20deployment%20to%20one%20of%20the%20services%20which%20was%20impacting%20alerts%20view%20in%20the%20portal.%20Engineers%20continue%20to%20work%20on%20rolling%20back%20to%20healthy%20state.%3C%2FSPAN%3E%20Some%20customers%20may%20experience%20issues%20viewing%20alerts.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2001%2F16%2006%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Anupama%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Thursday%2C%2016%20January%202020%2000%3A15%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3ERoot%20cause%20has%20been%20isolated%20to%20recent%20deployment%20to%20one%20of%20the%20services%20which%20was%20impacting%20alerts%20view%20in%20the%20portal.%20To%20address%20this%20issue%20engineers%20are%20working%20on%20rolling%20back%20the%20service%20to%20previous%20healthy%20state.%20Some%20customers%20may%20experience%20issues%20viewing%20alerts%20and%20we%20estimate%203%20more%20hours%20before%20all%20alerts%20interface%20issues%20are%20addressed.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2001%2F16%2003%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Anupama%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Wednesday%2C%2015%20January%202020%2022%3A22%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20continue%20to%20investigate%20issues%20with%20alerts%20view%20in%20the%20portal.%20Preliminary%20investigation%20shows%20that%20a%20recent%20deployment%20to%20one%20of%20the%20services%20prevented%20users%20from%20viewing%20alerts%20in%20the%20portal.%20Customers%20continue%20to%20experience%20issues%20with%20viewing%20alerts%20in%20portal.%20We%20are%20working%20to%20establish%20the%20start%20time%20for%20the%20issue%2C%20initial%20findings%20indicate%20that%20the%20problem%20began%20at%2001%2F10%20~17%3A30%20UTC.%20We%20currently%20have%20no%20estimate%20for%20resolution.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2001%2F16%2000%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Anupama%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Wednesday%2C%2015%20January%202020%2020%3A16%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%26nbsp%3B%20Azure%20monitor%20Alerts%20and%20are%20actively%20investigating.%20Some%20customers%20may%20be%20experiencing%20issues%20while%20viewing%20alerts%20in%20Azure%20portal.%26nbsp%3B%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20none%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2001%2F15%2022%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Anupama%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1111100%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EActivity%20Log%20Alerts%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ELog%20Alerts%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ELog%20Search%20Alerts%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EMetric%20Alerts%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ESmart%20Diagnostics%20Alerts%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Thursday, 16 January 2020 08:09 UTC

We've confirmed that all systems are back to normal with no customer impact as of 01/16, 08:00 UTC. Our logs show the incident started on 01/11, 04:25 UTC and that during the 5 days 3 hours 35 minutes that it took to resolve the issue some of customers in US Gov and Azure China Region might have experienced issue while viewing alerts.
  • Root Cause: The failure was due to an issue with one of our dependent service. 
  • Incident Timeline: 5 Days, 3 Hours & 35 minutes - 01/11, 04:25 UTC through 01/16, 08:00 UTC
We understand that customers rely on Azure Monitor as a critical service and apologize for any impact this incident caused.

-Monish

Update: Thursday, 16 January 2020 05:32 UTC 
The issue got mitigated for the customers in US Gov Region as of 01/16 04:25 UTC and the engineers are continuing to work on rolling back the service Azure China for mitigating the issue. Some customer in Azure China Region might still experience issue in viewing the alerts.
  • Work Around: None
  • Next Update: Before 01/16 10:00 UTC
-Monish

Update: Thursday, 16 January 2020 02:38 UTC

Root cause has been isolated to recent deployment to one of the services which was impacting alerts view in the portal. Engineers continue to work on rolling back to healthy state. Some customers may experience issues viewing alerts.
  • Work Around: None
  • Next Update: Before 01/16 06:00 UTC
-Anupama

Update: Thursday, 16 January 2020 00:15 UTC

Root cause has been isolated to recent deployment to one of the services which was impacting alerts view in the portal. To address this issue engineers are working on rolling back the service to previous healthy state. Some customers may experience issues viewing alerts and we estimate 3 more hours before all alerts interface issues are addressed.
  • Work Around: None
  • Next Update: Before 01/16 03:30 UTC
-Anupama

Update: Wednesday, 15 January 2020 22:22 UTC

We continue to investigate issues with alerts view in the portal. Preliminary investigation shows that a recent deployment to one of the services prevented users from viewing alerts in the portal. Customers continue to experience issues with viewing alerts in portal. We are working to establish the start time for the issue, initial findings indicate that the problem began at 01/10 ~17:30 UTC. We currently have no estimate for resolution.
  • Work Around: None
  • Next Update: Before 01/16 00:30 UTC
-Anupama

Initial Update: Wednesday, 15 January 2020 20:16 UTC

We are aware of issues within  Azure monitor Alerts and are actively investigating. Some customers may be experiencing issues while viewing alerts in Azure portal. 
  • Work Around: none
  • Next Update: Before 01/15 22:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Anupama