Home
%3CLINGO-SUB%20id%3D%22lingo-sub-1086440%22%20slang%3D%22en-US%22%3EExperiencing%20Alerting%20failure%20for%20Log%20Search%20Alerts%20in%20South%20UK%20region%20-%2001%2F02%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1086440%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Friday%2C%2003%20January%202020%2017%3A00%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20as%20of%2001%2F03%2C%2016%3A37%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2001%2F02%2C%2021%3A08%20UTC%20and%20that%20during%20the%26nbsp%3B%20hours%20that%20it%20took%20to%20resolve%20the%20issue%20some%20customers%20experienced%20loss%20of%20Log%20alerts%20being%20delivered.%3CBR%20%2F%3E%3CUL%3E%0A%20%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20backlog%20buildup%20that%20was%20caused%20by%20timeout%20to%20reach%20a%20specific%20storage%20account.%3C%2FLI%3E%0A%20%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%26nbsp%3B%2019%20Hours%20%26amp%3B%2029%20minutes%20-%2001%2F02%2C%2021%3A08%20UTC%20through%2001%2F03%2C%2016%3A37%20UTC%3C%2FLI%3E%0A%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Log%20Search%20Alerts%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-chandar%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Friday%2C%2003%20January%202020%2015%3A50%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20continue%20to%20investigate%20issues%20within%20Log%20Search%20Alerts.%20Root%20cause%20is%20not%20fully%20understood%20at%20this%20time.%26nbsp%3BCustomers%20continue%20to%20experience%20missing%20Log%20Alerts.%20We%20currently%20have%20no%20estimate%20for%20resolution..%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%26nbsp%3B%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2001%2F03%2020%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Madhuri%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Friday%2C%2003%20January%202020%2009%3A47%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20continue%20to%20investigate%20issues%20within%20Log%20Search%20Alerts.%20Root%20cause%20is%20not%20fully%20understood%20at%20this%20time.%20Customers%20continue%20to%20experience%20missing%20Log%20Alerts.%20We%20currently%20have%20no%20estimate%20for%20resolution.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%26nbsp%3B%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2001%2F03%2016%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Madhuri%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Friday%2C%2003%20January%202020%2005%3A39%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20continue%20to%20investigate%20issues%20within%20Log%20Search%20Alerts.%20Root%20cause%20is%20not%20fully%20understood%20at%20this%20time.%20Customers%20continue%20to%20experience%20missing%20Log%20Alerts.%20We%20currently%20have%20no%20estimate%20for%20resolution.%26nbsp%3B%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%26nbsp%3B%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2001%2F03%2010%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Madhuri%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Friday%2C%2003%20January%202020%2003%3A44%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20continue%20to%20investigate%20issues%20within%20Log%20Search%20Alerts.%20Root%20cause%20is%20not%20fully%20understood%20at%20this%20time.%20%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20rgb(255%2C%20255%2C%20255)%3B%20color%3A%20rgb(0%2C%200%2C%200)%3B%20font-family%3A%20%22%20helvetica%3D%22%22%20neue%3D%22%22%3ECustomers%20continue%20to%20experience%20missing%20Log%20Alerts.%3C%2FSPAN%3E%20We%20are%20working%20to%20redeploy%20our%20components.%26nbsp%3B%20We%20currently%20have%20no%20estimate%20for%20resolution.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%3CNONE%20or%3D%22%22%20details%3D%22%22%3E%3C%2FNONE%3E%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2001%2F03%2006%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-chandar%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Friday%2C%2003%20January%202020%2000%3A43%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20continue%20to%20investigate%20issues%20within%20Log%20Search%20Alerts.%20Root%20cause%20is%20not%20fully%20understood%20at%20this%20time.%26nbsp%3B%20Customers%20continue%20to%20experience%20missing%20Log%20Alerts.%20We%20are%20working%20to%20establish%20the%20start%20time%20for%20the%20issue%2C%20initial%20findings%20indicate%20that%20the%20problem%20began%20at%2002-Jan-2020%2021%3A08.%20We%20currently%20have%20no%20estimate%20for%20resolution.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%3CNONE%20or%3D%22%22%20details%3D%22%22%3E%3C%2FNONE%3E%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2001%2F03%2003%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-chandar%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CFONT%20face%3D%22%22%20helvetica%3D%22%22%20neue%3D%22%22%3EInitial%20Update%3A%20Thursday%2C%2002%20January%202020%2022%3A16%20UTC%3C%2FFONT%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CFONT%20face%3D%22%22%20helvetica%3D%22%22%20neue%3D%22%22%3E%3CBR%20%2F%3E%3C%2FFONT%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CFONT%20face%3D%22%22%20helvetica%3D%22%22%20neue%3D%22%22%3EWe%20are%20aware%20of%20issues%20within%20Log%20Search%20Alerts%20and%20are%20actively%20investigating.%20Customers%20may%20not%20be%20receiving%20log%20Alerts%20at%20this%20time.%3C%2FFONT%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3CFONT%20face%3D%22%22%20helvetica%3D%22%22%20neue%3D%22%22%3E%3CBR%20%2F%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20Work%20Around%3A%20none.%3CBR%20%2F%3E%26nbsp%3B%20%26nbsp%3B%26nbsp%3B%20Next%20Update%3A%20Before%2001%2F03%2000%3A30%20UTC%20%3CBR%20%2F%3E%3C%2FFONT%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CFONT%20face%3D%22%22%20helvetica%3D%22%22%20neue%3D%22%22%3E%3CBR%20%2F%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.-Chandar%3CBR%20%2F%3E%3C%2FFONT%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1086440%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3ELog%20Search%20Alerts%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Friday, 03 January 2020 17:00 UTC

We've confirmed that all systems are back to normal as of 01/03, 16:37 UTC. Our logs show the incident started on 01/02, 21:08 UTC and that during the  hours that it took to resolve the issue some customers experienced loss of Log alerts being delivered.
  • Root Cause: The failure was due to backlog buildup that was caused by timeout to reach a specific storage account.
  • Incident Timeline:  19 Hours & 29 minutes - 01/02, 21:08 UTC through 01/03, 16:37 UTC
We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused.

-chandar

Update: Friday, 03 January 2020 15:50 UTC

We continue to investigate issues within Log Search Alerts. Root cause is not fully understood at this time. Customers continue to experience missing Log Alerts. We currently have no estimate for resolution..
  • Work Around
  • Next Update: Before 01/03 20:00 UTC
-Madhuri

Update: Friday, 03 January 2020 09:47 UTC

We continue to investigate issues within Log Search Alerts. Root cause is not fully understood at this time. Customers continue to experience missing Log Alerts. We currently have no estimate for resolution.
  • Work Around
  • Next Update: Before 01/03 16:00 UTC
-Madhuri

Update: Friday, 03 January 2020 05:39 UTC

We continue to investigate issues within Log Search Alerts. Root cause is not fully understood at this time. Customers continue to experience missing Log Alerts. We currently have no estimate for resolution. 
  • Work Around
  • Next Update: Before 01/03 10:00 UTC
-Madhuri

Update: Friday, 03 January 2020 03:44 UTC

We continue to investigate issues within Log Search Alerts. Root cause is not fully understood at this time. Customers continue to experience missing Log Alerts. We are working to redeploy our components.  We currently have no estimate for resolution.
  • Work Around:
  • Next Update: Before 01/03 06:00 UTC
-chandar

Update: Friday, 03 January 2020 00:43 UTC

We continue to investigate issues within Log Search Alerts. Root cause is not fully understood at this time.  Customers continue to experience missing Log Alerts. We are working to establish the start time for the issue, initial findings indicate that the problem began at 02-Jan-2020 21:08. We currently have no estimate for resolution.
  • Work Around:
  • Next Update: Before 01/03 03:00 UTC
-chandar

Initial Update: Thursday, 02 January 2020 22:16 UTC

We are aware of issues within Log Search Alerts and are actively investigating. Customers may not be receiving log Alerts at this time.    
      Work Around: none.
     Next Update: Before 01/03 00:30 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Chandar