%3CLINGO-SUB%20id%3D%22lingo-sub-942639%22%20slang%3D%22en-US%22%3EExperiencing%20ARM%20Deployment%20Failure%20for%20Log%20Search%20Alerts%20-%2010%2F24%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-942639%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Thursday%2C%2024%20October%202019%2010%3A41%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2010%2F24%2C%2008%3A30%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2010%2F23%2C%2011%3A50%20UTC%20and%20that%20during%20the%2020%20hours%2040%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%20some%20of%20customers%20experienced%20failure%20in%20updating%20the%20rules%20that%20was%20created%20within%20the%20impact%20duration.%3CBR%20%2F%3E%3CUL%3E%0A%20%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20an%20issue%20with%20one%20of%20our%20dependent%20service%3C%2FLI%3E%0A%20%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%2020%20Hours%20%26amp%3B%2040%20minutes%20-%2010%2F23%2C%2011%3A50%20UTC%20through%2010%2F24%2C%2008%3A30%20UTC%3C%2FLI%3E%0A%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Log%20Search%20Alerts%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Monish%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Thursday%2C%2024%20October%202019%2008%3A52%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Log%20Search%20Alerts%20and%20are%20actively%20investigating.%20Some%20customers%20may%20experience%20failures%20when%20deploying%20through%20ARM.%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2010%2F24%2013%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Monish%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-942639%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3ELog%20Search%20Alerts%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Thursday, 24 October 2019 10:41 UTC

We've confirmed that all systems are back to normal with no customer impact as of 10/24, 08:30 UTC. Our logs show the incident started on 10/23, 11:50 UTC and that during the 20 hours 40 minutes that it took to resolve the issue some of customers experienced failure in updating the rules that was created within the impact duration.
  • Root Cause: The failure was due to an issue with one of our dependent service
  • Incident Timeline: 20 Hours & 40 minutes - 10/23, 11:50 UTC through 10/24, 08:30 UTC
We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused.

-Monish

Initial Update: Thursday, 24 October 2019 08:52 UTC

We are aware of issues within Log Search Alerts and are actively investigating. Some customers may experience failures when deploying through ARM.
  • Next Update: Before 10/24 13:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Monish