Experiencing Alerting failure for Log Analytics Metric Alerts - 10/25 - Resolved

Published Oct 24 2019 09:47 PM 959 Views
Final Update: Friday, 25 October 2019 12:05 UTC

We've confirmed that all systems are back to normal with no customer impact as of 10/25, 11:20 UTC. Our logs show the incident started on 10/25, 03:35 UTC and that during the 7 hours and 45 minutes that it took to resolve the issue Customers experienced alerting failure from Log Analytics Metric alerts in East US region.
  • Root Cause: The failure was due to issue with our backend service.
  • Incident Timeline: 7 Hours & 45 minutes - 10/25, 03:35 UTC through 10/25, 11:20 UTC
We understand that customers rely on Metric Alerts as a critical service and apologize for any impact this incident caused.

-Rama

Update: Friday, 25 October 2019 07:47 UTC

We continue to investigate issues within Metric Alerts. Root cause is not fully understood at this time. Some customers continue to experience alerting failure from Log Analytics Metric alerts in East US region. We are working to establish the start time for the issue, initial findings indicate that the problem began at 10/25 ~03:35 UTC. We currently have no estimate for resolution.
  • Work Around: None
  • Next Update: Before 10/25 12:00 UTC
-Rama

Initial Update: Friday, 25 October 2019 04:43 UTC

We are aware of issues within Log Analytics Metric Alerts and are actively investigating. Customers may experience alerting failure from Log Analytics Metric alerts in East US region .
  • Work Around: None
  • Next Update: Before 10/25 07:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Rama

%3CLINGO-SUB%20id%3D%22lingo-sub-951241%22%20slang%3D%22en-US%22%3EExperiencing%20Alerting%20failure%20for%20Log%20Analytics%20Metric%20Alerts%20-%2010%2F25%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-951241%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Friday%2C%2025%20October%202019%2012%3A05%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2010%2F25%2C%2011%3A20%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2010%2F25%2C%2003%3A35%20UTC%20and%20that%20during%20the%207%20hours%20and%2045%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%20Customers%20experienced%20alerting%20failure%20from%20Log%20Analytics%20Metric%20alerts%20in%20East%20US%20region.%3CBR%20%2F%3E%3CUL%3E%0A%20%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20issue%20with%20our%20backend%20service.%3C%2FLI%3E%0A%20%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%207%20Hours%20%26amp%3B%2045%20minutes%20-%2010%2F25%2C%2003%3A35%20UTC%20through%2010%2F25%2C%2011%3A20%20UTC%3C%2FLI%3E%0A%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Metric%20Alerts%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Rama%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Friday%2C%2025%20October%202019%2007%3A47%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20continue%20to%20investigate%20issues%20within%20Metric%20Alerts.%20Root%20cause%20is%20not%20fully%20understood%20at%20this%20time.%20Some%20customers%20continue%20to%20experience%20alerting%20failure%20from%20Log%20Analytics%20Metric%20alerts%20in%20East%20US%20region.%20We%20are%20working%20to%20establish%20the%20start%20time%20for%20the%20issue%2C%20initial%20findings%20indicate%20that%20the%20problem%20began%20at%2010%2F25%20~03%3A35%20UTC.%20We%20currently%20have%20no%20estimate%20for%20resolution.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2010%2F25%2012%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Rama%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Friday%2C%2025%20October%202019%2004%3A43%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Log%20Analytics%20Metric%20Alerts%20and%20are%20actively%20investigating.%20Customers%20may%20experience%20alerting%20failure%20from%26nbsp%3BLog%20Analytics%20Metric%20alerts%20in%20East%20US%20region%20.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2010%2F25%2007%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Rama%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-951241%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EMetric%20Alerts%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Version history
Last update:
‎Oct 25 2019 05:11 AM
Updated by: