%3CLINGO-SUB%20id%3D%22lingo-sub-1325325%22%20slang%3D%22en-US%22%3EExperiencing%20Alerting%20failure%20for%20Log%20Search%20Alerts%20in%20North%20Europe%20-%2004%2F21%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1325325%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Tuesday%2C%2021%20April%202020%2023%3A55%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20as%20of%2004%2F21%2C%2023%3A55%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2004%2F21%2C11%3A50%20UTC%20and%20that%20during%20the%2011%20hours%2057minutes%20that%20it%20took%20to%20resolve%20the%20issue%20some%20customers%20in%20North%20Europe%20region%20may%20not%20have%20received%20Log%20Search%20Alerts.%26nbsp%3B%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20issue%20in%20one%20of%20our%20dependent%20services.%3C%2FLI%3E%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%2011%20Hours%20%26amp%3B%2057%20minutes%20-%2004%2F21%2C%2011%3A50%20UTC%20through%2004%2F21%2C%2023%3A47%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Log%20Search%20Alerts%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Sindhu%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Tuesday%2C%2021%20April%202020%2023%3A11%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20continue%20to%20investigate%20issues%20within%20Log%20Search%20Alerts.%20Root%20cause%20is%20not%20fully%20understood%20at%20this%20time.%20Some%20customers%20continue%20to%20experience%20alerting%20failures%20in%20North%20Europe%20region.%20Initial%20findings%20indicate%20that%20the%20problem%20began%20at%2004%2F21%20~11.50%20UTC.%20We%20currently%20have%20no%20estimate%20for%20resolution.%26nbsp%3B%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%26nbsp%3B%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2004%2F22%2002%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Sindhu%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Tuesday%2C%2021%20April%202020%2020%3A01%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20continue%20to%20investigate%20issues%20within%20Log%20Search%20Alerts.%20Root%20cause%20is%20not%20fully%20understood%20at%20this%20time.%20Some%20customers%20continue%20to%20experience%20alerting%20failures%20in%20North%20Europe%20region.%20Initial%20findings%20indicate%20that%20the%20problem%20began%20at%2004%2F21%20~11.50%20UTC.%20We%20currently%20have%20no%20estimate%20for%20resolution.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2004%2F21%2023%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Sindhu%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Tuesday%2C%2021%20April%202020%2017%3A53%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20continue%20to%20investigate%20issues%20within%20Log%20Search%20Alerts.%20Root%20cause%20is%20not%20fully%20understood%20at%20this%20time.%20Some%20customers%20will%20continue%20to%20experience%20alerting%20failures%20in%20North%20Europe%20region.%20Initial%20findings%20indicate%20that%20the%20problem%20began%20at%2004%2F21%20~11.50%20UTC.%20We%20currently%20have%20no%20estimate%20for%20resolution.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2004%2F21%2020%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Sindhu%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Tuesday%2C%2021%20April%202020%2015%3A53%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Log%20Search%20Alerts%20and%20are%20actively%20investigating.%20Some%20customers%20may%20experience%20Alerting%20failure%20in%20North%20Europe%20region.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2004%2F21%2018%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Madhuri%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1325325%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3ELog%20Search%20Alerts%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Tuesday, 21 April 2020 23:55 UTC

We've confirmed that all systems are back to normal as of 04/21, 23:55 UTC. Our logs show the incident started on 04/21,11:50 UTC and that during the 11 hours 57minutes that it took to resolve the issue some customers in North Europe region may not have received Log Search Alerts. 
  • Root Cause: The failure was due to issue in one of our dependent services.
  • Incident Timeline: 11 Hours & 57 minutes - 04/21, 11:50 UTC through 04/21, 23:47 UTC
We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused.

-Sindhu

Update: Tuesday, 21 April 2020 23:11 UTC

We continue to investigate issues within Log Search Alerts. Root cause is not fully understood at this time. Some customers continue to experience alerting failures in North Europe region. Initial findings indicate that the problem began at 04/21 ~11.50 UTC. We currently have no estimate for resolution. 
  • Work Around: None 
  • Next Update: Before 04/22 02:30 UTC
-Sindhu

Update: Tuesday, 21 April 2020 20:01 UTC

We continue to investigate issues within Log Search Alerts. Root cause is not fully understood at this time. Some customers continue to experience alerting failures in North Europe region. Initial findings indicate that the problem began at 04/21 ~11.50 UTC. We currently have no estimate for resolution.
  • Work Around: None
  • Next Update: Before 04/21 23:30 UTC
-Sindhu

Update: Tuesday, 21 April 2020 17:53 UTC

We continue to investigate issues within Log Search Alerts. Root cause is not fully understood at this time. Some customers will continue to experience alerting failures in North Europe region. Initial findings indicate that the problem began at 04/21 ~11.50 UTC. We currently have no estimate for resolution.
  • Work Around: None
  • Next Update: Before 04/21 20:00 UTC
-Sindhu

Initial Update: Tuesday, 21 April 2020 15:53 UTC

We are aware of issues within Log Search Alerts and are actively investigating. Some customers may experience Alerting failure in North Europe region.
  • Work Around: None
  • Next Update: Before 04/21 18:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Madhuri