%3CLINGO-SUB%20id%3D%22lingo-sub-934209%22%20slang%3D%22en-US%22%3EExperiencing%20Alerting%20failure%20for%20Failure%20Anomalies%20Alerts%20-%2010%2F23%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-934209%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Wednesday%2C%2023%20October%202019%2010%3A42%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2010%2F23%2C%2009%3A40%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2010%2F23%2C%2008%3A20%20UTC%20and%20that%20during%20the%201%20hour%2020%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%20Customers%20would%20have%20experienced%20alerting%20failure%20from%20Failure%20Anomalies%20alerts%20configured%20in%26nbsp%3B%20South%20Central%20US%2C%20South%20East%20Asia%20and%20West%20US%20region.%3CBR%20%2F%3E%3CUL%3E%0A%20%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20issue%20with%20one%20of%20our%20backend%20services.%3C%2FLI%3E%0A%20%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%201%20Hours%20%26amp%3B%2020%20minutes%20-%2010%2F23%2C%2008%3A20%20UTC%20through%2010%2F23%2C%2009%3A40%20UTC%3C%2FLI%3E%0A%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Log%20Search%20Alerts%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Rama%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-934209%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3ELog%20Search%20Alerts%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Wednesday, 23 October 2019 10:42 UTC

We've confirmed that all systems are back to normal with no customer impact as of 10/23, 09:40 UTC. Our logs show the incident started on 10/23, 08:20 UTC and that during the 1 hour 20 minutes that it took to resolve the issue Customers would have experienced alerting failure from Failure Anomalies alerts configured in  South Central US, South East Asia and West US region.
  • Root Cause: The failure was due to issue with one of our backend services.
  • Incident Timeline: 1 Hours & 20 minutes - 10/23, 08:20 UTC through 10/23, 09:40 UTC
We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused.

-Rama