%3CLINGO-SUB%20id%3D%22lingo-sub-871309%22%20slang%3D%22en-US%22%3EExperiencing%20Alerting%20failure%20for%20Log%20Search%20Alerts%20in%20East%20US%20Region%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-871309%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Tuesday%2C%2024%20September%202019%2008%3A21%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2009%2F24%2C%2011%3A30%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2009%2F24%2C%2001%3A00%20UTC%20and%20that%20during%20the%2010%20hours%20%26amp%3B%2030%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%2040%25%20of%20customers%20experienced%20alert%20delivering%20latency%20in%20East%20US%20region.%3CBR%20%2F%3E%3CUL%3E%0A%20%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20an%20issue%20in%20one%20of%20our%20back-end%20service.%3C%2FLI%3E%0A%20%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%2010%20Hours%20%26amp%3B%2030%20minutes%20-%2009%2F24%2C%2001%3A00%20UTC%20through%2009%2F24%2C%2011%3A30%20UTC%3C%2FLI%3E%0A%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Log%20Search%20Alerts%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Naresh%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Tuesday%2C%2024%20September%202019%2006%3A29%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20continue%20to%20investigate%20issues%20within%20Log%20Search%20Alerts.%20Root%20cause%20is%20not%20fully%20understood%20at%20this%20time.%20Some%20customers%20continue%20to%20experience%20issues%20with%20alerts%20not%20being%20delivered%20in%20East%20US%20region.%20We%20currently%20have%20no%20estimate%20for%20resolution.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2009%2F24%2009%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Naresh%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Tuesday%2C%2024%20September%202019%2002%3A00%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Log%20Search%20Alerts%20in%20East%20US%20Region%20and%20are%20actively%20investigating.%20Some%20customers%20may%20experience%20issue%20with%20alerts%20not%20being%20delivered.%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2009%2F24%2006%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Jayadev%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-871309%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3ELog%20Search%20Alerts%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Tuesday, 24 September 2019 08:21 UTC

We've confirmed that all systems are back to normal with no customer impact as of 09/24, 11:30 UTC. Our logs show the incident started on 09/24, 01:00 UTC and that during the 10 hours & 30 minutes that it took to resolve the issue 40% of customers experienced alert delivering latency in East US region.
  • Root Cause: The failure was due to an issue in one of our back-end service.
  • Incident Timeline: 10 Hours & 30 minutes - 09/24, 01:00 UTC through 09/24, 11:30 UTC
We understand that customers rely on Log Search Alerts as a critical service and apologize for any impact this incident caused.

-Naresh

Update: Tuesday, 24 September 2019 06:29 UTC

We continue to investigate issues within Log Search Alerts. Root cause is not fully understood at this time. Some customers continue to experience issues with alerts not being delivered in East US region. We currently have no estimate for resolution.
  • Work Around: None
  • Next Update: Before 09/24 09:30 UTC
-Naresh

Initial Update: Tuesday, 24 September 2019 02:00 UTC

We are aware of issues within Log Search Alerts in East US Region and are actively investigating. Some customers may experience issue with alerts not being delivered.
  • Next Update: Before 09/24 06:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Jayadev