Home
%3CLINGO-SUB%20id%3D%22lingo-sub-873762%22%20slang%3D%22en-US%22%3EExperiencing%20Alerting%20failure%20issue%20in%20Azure%20Portal%20for%20Many%20Data%20Types%20-%2009%2F25%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-873762%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Wednesday%2C%2025%20September%202019%2019%3A25%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2009%2F25%2C%2019%3A15%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2009%2F11%2C16%3A00%20UTC%20and%20that%20during%20the%2014%20days%2C3%20hours%20and%2015%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%20some%20of%20the%20customers%20may%20have%20experienced%20Alerting%20failure%20for%20Exception%20based%20metric%20alerts%20in%20Application%20Insights.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20deployment%20in%20one%20of%20the%20backend%20services.%3C%2FLI%3E%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%20%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20rgb(255%2C%20255%2C%20255)%3B%20color%3A%20rgb(0%2C%200%2C%200)%3B%20font-family%3A%20%26quot%3BHelvetica%20Neue%26quot%3B%2CHelvetica%2CArial%2Csans-serif%3B%20font-size%3A%2014px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20400%3B%20letter-spacing%3A%20normal%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3E14%20days%2C3%20hours%20and%2015%20minutes%3C%2FSPAN%3E%20-%2009%2F11%2C%2016%3A00%20UTC%20through%2009%2F25%2C%20%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20rgb(255%2C%20255%2C%20255)%3B%20color%3A%20rgb(0%2C%200%2C%200)%3B%20font-family%3A%20%26quot%3BHelvetica%20Neue%26quot%3B%2CHelvetica%2CArial%2Csans-serif%3B%20font-size%3A%2014px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20400%3B%20letter-spacing%3A%20normal%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3E19%3A15%3C%2FSPAN%3E%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Application%20Insights%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Jayadev%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Wednesday%2C%2025%20September%202019%2017%3A49%20UTC%3CBR%20%2F%3E%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20rgb(255%2C%20255%2C%20255)%3B%20color%3A%20rgb(0%2C%200%2C%200)%3B%20font-family%3A%20%26quot%3BHelvetica%20Neue%26quot%3B%2CHelvetica%2CArial%2Csans-serif%3B%20font-size%3A%2014px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20400%3B%20letter-spacing%3A%20normal%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3E%3CBR%20%2F%3E%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20rgb(255%2C%20255%2C%20255)%3B%20color%3A%20rgb(0%2C%200%2C%200)%3B%20font-family%3A%20%26quot%3BHelvetica%20Neue%26quot%3B%2CHelvetica%2CArial%2Csans-serif%3B%20font-size%3A%2014px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20400%3B%20letter-spacing%3A%20normal%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3EThe%20fix%20had%20been%20deployed%20to%2016%20of%2018%20regions.%20East%20US%20and%20West%20Europe%20are%20two%20regions%20to%20be%20deployed%20further.%20Customers%20in%20the%20remaining%20regions%20will%20not%20experience%20Alerting%20failure%20and%20the%20metrics%20will%20work%20as%20expected.%3C%2FSPAN%3E%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2009%2F25%2022%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Jayadev%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Wednesday%2C%2025%20September%202019%2013%3A33%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EThe%20fix%20had%20been%20deployed%20to%2015%26nbsp%3B%20of%2018%20regions%2C%20East%20US%2C%20West%20Europe%20and%20South%20Central%20US%20are%20three%20region%20to%20be%20deployed%20further.%20Customers%20in%20the%20remaining%20regions%20will%20not%20experience%20Alerting%20failure%20and%20the%20metrics%20will%20work%20as%20expected.%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2009%2F25%2019%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Monish%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Wednesday%2C%2025%20September%202019%2010%3A35%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3ERoot%20cause%20has%20been%20isolated%20to%20Alerting%20Failure%20for%20Exception%20based%20metric%20which%20was%20impacting%20the%20alert%20been%20resolved%20and%20generating%20new%20alerts.%26nbsp%3B%20Engineers%20are%20involved%20in%20fixing%20the%20issue%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2009%2F25%2015%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Monish%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Wednesday%2C%2025%20September%202019%2006%3A17%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Application%20Insights%20and%20are%20actively%20investigating.%20Some%20customers%20may%20experience%20Alerting%20failure%20for%20Exception%20metrics%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2009%2F25%2010%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Monish%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-873762%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EApplication%20Insights%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Wednesday, 25 September 2019 19:25 UTC

We've confirmed that all systems are back to normal with no customer impact as of 09/25, 19:15 UTC. Our logs show the incident started on 09/11,16:00 UTC and that during the 14 days,3 hours and 15 minutes that it took to resolve the issue some of the customers may have experienced Alerting failure for Exception based metric alerts in Application Insights.
  • Root Cause: The failure was due to deployment in one of the backend services.
  • Incident Timeline: 14 days,3 hours and 15 minutes - 09/11, 16:00 UTC through 09/25, 19:15 UTC
We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Jayadev

Update: Wednesday, 25 September 2019 17:49 UTC

The fix had been deployed to 16 of 18 regions. East US and West Europe are two regions to be deployed further. Customers in the remaining regions will not experience Alerting failure and the metrics will work as expected.
  • Next Update: Before 09/25 22:00 UTC
-Jayadev

Update: Wednesday, 25 September 2019 13:33 UTC

The fix had been deployed to 15  of 18 regions, East US, West Europe and South Central US are three region to be deployed further. Customers in the remaining regions will not experience Alerting failure and the metrics will work as expected.
  • Next Update: Before 09/25 19:00 UTC
-Monish

Update: Wednesday, 25 September 2019 10:35 UTC

Root cause has been isolated to Alerting Failure for Exception based metric which was impacting the alert been resolved and generating new alerts.  Engineers are involved in fixing the issue
  • Next Update: Before 09/25 15:00 UTC
-Monish

Initial Update: Wednesday, 25 September 2019 06:17 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may experience Alerting failure for Exception metrics
  • Next Update: Before 09/25 10:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Monish