Experiencing Alerting Latency for Activity Log Alerts - 04/29 - Resolved

Published Apr 29 2022 12:52 PM 746 Views
Final Update: Friday, 29 April 2022 22:37 UTC

We've confirmed that all systems are back to normal with no customer impact as of 04/29, 21:30 UTC. Our logs show the incident started on 04/29, 14:00 UTC and that during the 7 hours and 30 minutes that it took to resolve the issue some of the customers experienced latency in getting notifications from Activity Log Alerts.
  • Root Cause: A backend service had reached an operational threshold causing notification latency to occur.
  • Incident Timeline: 7 Hours & 30 minutes - 04/29, 14:00 UTC through 04/29, 21:30 UTC
We understand that customers rely on Activity Log Alerts as a critical service and apologize for any impact this incident caused.

-Saika

Update: Friday, 29 April 2022 19:40 UTC

Root cause has been isolated to sudden spike in large number of events overloaded the machines causing the delay in Activity Log Alerts. To address this issue, we scaled out the number of instances of this service. Some customers may experience latency in getting notifications from Activity Log Alerts. 
  • Work Around: None
  • Next Update: Before 04/29 22:00 UTC
-Rahul Prasanna

%3CLINGO-SUB%20id%3D%22lingo-sub-3299125%22%20slang%3D%22en-US%22%3EExperiencing%20Alerting%20Latency%20for%20Activity%20Log%20Alerts%20-%2004%2F29%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-3299125%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Friday%2C%2029%20April%202022%2022%3A37%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2004%2F29%2C%2021%3A30%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2004%2F29%2C%2014%3A00%20UTC%20and%20that%20during%20the%207%20hours%20and%2030%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%20some%20of%20the%20customers%20experienced%20latency%20in%20getting%20notifications%20from%20Activity%20Log%20Alerts.%3CBR%20%2F%3E%3CUL%3E%0A%20%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20A%20backend%20service%20had%20reached%20an%20operational%20threshold%20causing%20notification%20latency%20to%20occur.%3C%2FLI%3E%0A%20%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%207%20Hours%20%26amp%3B%2030%20minutes%20-%2004%2F29%2C%2014%3A00%20UTC%20through%2004%2F29%2C%2021%3A30%20UTC%3C%2FLI%3E%0A%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Activity%20Log%20Alerts%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E-Saika%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Friday%2C%2029%20April%202022%2019%3A40%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3ERoot%20cause%20has%20been%20isolated%20to%20sudden%20spike%20in%20large%20number%20of%20events%20overloaded%20the%20machines%20causing%20the%20delay%20in%20Activity%20Log%20Alerts.%20To%20address%20this%20issue%2C%20we%20scaled%20out%20the%20number%20of%20instances%20of%20this%20service.%20Some%20customers%20may%20experience%20latency%20in%20getting%20notifications%20from%20Activity%20Log%20Alerts.%26nbsp%3B%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2004%2F29%2022%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Rahul%20Prasanna%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E
Version history
Last update:
‎Apr 29 2022 03:48 PM
Updated by: