Home
%3CLINGO-SUB%20id%3D%22lingo-sub-877522%22%20slang%3D%22en-US%22%3EExperiencing%20Issues%20with%20chart%20loads%20for%20dynamic%20alerts%2009%2F26%20-%20Mitigating%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-877522%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Thursday%2C%2026%20September%202019%2023%3A42%20UTC%3CBR%20%2F%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CBR%20%2F%3ERoot%20cause%20has%20been%20identified%20as%20a%20code-related%20issue.To%20address%20this%20issue%20a%20hotfix%20has%20been%20tested%20and%20validated%20in%20the%20pre-production%20environment.Currently%20the%20build%20is%20being%20deployed%20to%20all%20the%20regions%20and%20we%20expect%20the%20deployment%20to%20be%20completed%20in%20approximately%2024%20hours.Customers%20will%20continue%20to%20experience%20issues%20with%20charts%20not%20being%20loaded%20for%20dynamic%20alerts%20based%20on%20metrics%20until%20the%20deployment%20is%20completed%20to%20all%20regions.%20The%20problem%20was%20found%20to%20have%20started%20at%2009%2F24%2020%3A00%20UTC.%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2009%2F27%2018%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Jayadev%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Thursday%2C%2026%20September%202019%2019%3A26%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Azure%20Monitor%20Services%20and%20are%20actively%20investigating.%20%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20rgb(255%2C%20255%2C%20255)%3B%20color%3A%20rgb(0%2C%200%2C%200)%3B%20font-family%3A%20%26quot%3BHelvetica%20Neue%26quot%3B%2CHelvetica%2CArial%2Csans-serif%3B%20font-size%3A%2014px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20400%3B%20letter-spacing%3A%20normal%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3ECustomers%20may%20experience%20issues%20with%20charts%20not%20being%20loaded%20for%20dynamic%20alerts%20based%20on%20metrics%20and%20the%20start%20time%20for%20the%20issue%20has%20been%20revised%20to%20be%20from%2009%2F24%2020%3A00%20UTC.%3C%2FSPAN%3E%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2009%2F26%2023%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Jayadev%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Thursday%2C%2026%20September%202019%2018%3A02%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20the%20issues%20within%20Azure%20Monitor%20Services%20and%20are%20actively%20investigating.Customers%20may%20experience%20issues%20with%20charts%20not%20being%20loaded%20for%20dynamic%20alerts%20based%20on%20metrics.%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2009%2F26%2022%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Jayadev%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-877522%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EMetric%20Alerts%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Saturday, 28 September 2019 22:23 UTC

We've confirmed that all systems are back to normal with no customer impact as of 09/28, 22:00 UTC.Our logs show the incident started on 09/24, 20:00 UTC and that during the 4 days and 2 hours that it took to resolve the issue customers experienced issues with charts not being loaded for dynamic alerts based on metrics.
  • Root Cause: The failure was identified to be caused as part of recent deployment to our service.
  • Incident Timeline: 4 Days & 2 Hours - 09/24, 20:00 UTC through 09/28, 22:00 UTC
We understand that customers rely on Metric Alerts as a critical service and apologize for any impact this incident caused.

-Jayadev

Update: Saturday, 28 September 2019 20:22 UTC

We continue to deploy the fix to the affected regions. Customers in regions that have not updated will continue to experience issues with charts not being loaded for dynamic alerts based on metrics until the deployment is completed to those regions. The problem was found to have started at 09/24 20:00 UTC.
  • Next Update: Before 09/29 02:30 UTC
-Jayadev

Update: Saturday, 28 September 2019 14:32 UTC

We continue to deploy the fix to the affected regions. Customers in regions that have not updated will continue to experience issues with charts not being loaded for dynamic alerts based on metrics until the deployment is completed to those regions. The problem was found to have started at 09/24 20:00 UTC.
  • Work Around: None.
  • Next Update: Before 09/28 21:00 UTC
-Naresh

Update: Saturday, 28 September 2019 02:42 UTC

We continue to deploy the fix to the affected regions.Customers in regions that have not been updated will continue to experience issues with charts not being loaded for dynamic alerts based on metrics until the deployment is completed to those regions.The problem was found to have started at 09/24 20:00 UTC.
  • Next Update: Before 09/28 09:00 UTC
-Jayadev

Update: Friday, 27 September 2019 20:10 UTC

We continue to deploy the fix to the affected regions.Customers in regions that have not been updated will continue to experience issues with charts not being loaded for dynamic alerts based on metrics until the deployment is completed to those regions.The problem was found to have started at 09/24 20:00 UTC.
  • Next Update: Before 09/28 02:30 UTC
-Jayadev

Update: Thursday, 26 September 2019 23:42 UTC

Root cause has been identified as a code-related issue.To address this issue a hotfix has been tested and validated in the pre-production environment.Currently the build is being deployed to all the regions and we expect the deployment to be completed in approximately 24 hours.Customers will continue to experience issues with charts not being loaded for dynamic alerts based on metrics until the deployment is completed to all regions. The problem was found to have started at 09/24 20:00 UTC.
  • Next Update: Before 09/27 18:00 UTC
-Jayadev

Initial Update: Thursday, 26 September 2019 19:26 UTC

We are aware of issues within Azure Monitor Services and are actively investigating. Customers may experience issues with charts not being loaded for dynamic alerts based on metrics and the start time for the issue has been revised to be from 09/24 20:00 UTC.
  • Next Update: Before 09/26 23:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Jayadev

Initial Update: Thursday, 26 September 2019 18:02 UTC

We are aware of the issues within Azure Monitor Services and are actively investigating.Customers may experience issues with charts not being loaded for dynamic alerts based on metrics.
  • Next Update: Before 09/26 22:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Jayadev