Experiencing Data Latency for Log Analytics in East US 2 region - 09/04 - Resolved

Published Sep 04 2021 11:25 AM 1,046 Views
Final Update: Sunday, 05 September 2021 05:26 UTC

We've confirmed that all systems are back to normal with no customer impact as of 09/05, 04:40 UTC. Our logs show the incident started on 09/03, 20:30 UTC and that during the 32 hours and 10 minutes that it took to resolve the issue some customers may have experienced intermittent data latency and alert activation.
  • Root Cause: The failure was due to a backend dependency which became unhealthy.
  • Incident Timeline: 32 Hours & 10 minutes - 09/03, 20:30 UTC through 09/05, 04:40 UTC
We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused.

-Soumyajeet

Update: Sunday, 05 September 2021 04:04 UTC

Backlogged data continues to be drained. Customers will continue to see intermittent delayed data and incorrect alert activation. We expect that the remaining backlogged data will be completely ingested by 9/5 at 06:30 UTC.
  • Next Update: Before 09/05 07:00 UTC
-Jack Cantwell

Update: Sunday, 05 September 2021 01:04 UTC

The Log Analytics ingestion team has now fully recovered the problematic back end cluster and has scaled out even more in order to drain the queue of backlogged data as quickly as possible. Customers will continue to see intermittent delayed data and incorrect alert activation.
  • Next Update: Before 09/05 04:30 UTC
-Jack Cantwell

Update: Saturday, 04 September 2021 22:16 UTC

The Log Analytics ingestion team continues to mitigate the incident and the backlog in the data continues to drain. 

  • Next Update: Before 09/05 01:30 UTC
-Jack Cantwell

Update: Saturday, 04 September 2021 18:38 UTC

Root cause has been isolated to the failure of a back end compute cluster which caused incoming data to queue up and not be processed. To address this issue we restarted the cluster.  Ingestion is now working as expected and the backlog of data is now draining. However, because there is a large amount of backlogged data, some customers will continue to experience data latency and incorrect alert activation. We estimate several hours before all the backlogged data has been ingested.
  • Next Update: Before 09/04 22:00 UTC
-Jack Cantwell

Initial Update: Saturday, 04 September 2021 18:24 UTC

We are aware of issues within Log Analytics and are actively investigating. Some customers may experience delayed or missed Log Search Alerts and temporary unavailability of data.
  • Next Update: Before 09/04 19:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Jack Cantwell

%3CLINGO-SUB%20id%3D%22lingo-sub-2720452%22%20slang%3D%22en-US%22%3EExperiencing%20Data%20Latency%20for%20Log%20Analytics%20in%20East%20US%202%20region%20-%2009%2F04%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2720452%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Sunday%2C%2005%20September%202021%2005%3A26%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2009%2F05%2C%2004%3A40%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2009%2F03%2C%2020%3A30%20UTC%20and%20that%20during%20the%2032%20hours%20and%2010%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%20some%20customers%20may%20have%20experienced%20intermittent%20data%20latency%20and%20alert%20activation.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20a%20backend%20dependency%20which%20became%20unhealthy.%3C%2FLI%3E%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%2032%20Hours%20%26amp%3B%2010%20minutes%20-%2009%2F03%2C%2020%3A30%20UTC%20through%2009%2F05%2C%2004%3A40%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Azure%20Log%20Analytics%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Soumyajeet%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Sunday%2C%2005%20September%202021%2004%3A04%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EBacklogged%20data%20continues%20to%20be%20drained.%20Customers%20will%20continue%20to%20see%20intermittent%20delayed%20data%20and%20incorrect%20alert%20activation.%20We%20expect%20that%20the%20remaining%20backlogged%20data%20will%20be%20completely%20ingested%20by%209%2F5%20at%2006%3A30%20UTC.%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2009%2F05%2007%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Jack%20Cantwell%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Sunday%2C%2005%20September%202021%2001%3A04%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EThe%20Log%20Analytics%20ingestion%20team%20has%20now%20fully%20recovered%20the%20problematic%20back%20end%20cluster%20and%20has%20scaled%20out%20even%20more%20in%20order%20to%20drain%20the%20queue%20of%20backlogged%20data%20as%20quickly%20as%20possible.%20Customers%20will%20continue%20to%20see%20intermittent%20delayed%20data%20and%20incorrect%20alert%20activation.%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2009%2F05%2004%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Jack%20Cantwell%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Saturday%2C%2004%20September%202021%2022%3A16%20UTC%3CBR%20%2F%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CBR%20%2F%3E%3CP%3EThe%20Log%20Analytics%20ingestion%20team%20continues%20to%20mitigate%20the%20incident%20and%20the%20backlog%20in%20the%20data%20continues%20to%20drain.%26nbsp%3B%3C%2FP%3E%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2009%2F05%2001%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Jack%20Cantwell%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Saturday%2C%2004%20September%202021%2018%3A38%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3ERoot%20cause%20has%20been%20isolated%20to%20the%20failure%20of%20a%20back%20end%20compute%20cluster%20which%20caused%20incoming%20data%20to%20queue%20up%20and%20not%20be%20processed.%20To%20address%20this%20issue%20we%20restarted%20the%20cluster.%26nbsp%3B%20Ingestion%20is%20now%20working%20as%20expected%20and%20the%20backlog%20of%20data%20is%20now%20draining.%20However%2C%20because%20there%20is%20a%20large%20amount%20of%20backlogged%20data%2C%20some%20customers%20will%20continue%20to%20experience%20data%20latency%20and%20incorrect%20alert%20activation.%20We%20estimate%20several%20hours%20before%20all%20the%20backlogged%20data%20has%20been%20ingested.%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2009%2F04%2022%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Jack%20Cantwell%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Saturday%2C%2004%20September%202021%2018%3A24%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Log%20Analytics%20and%20are%20actively%20investigating.%20Some%20customers%20may%20experience%20delayed%20or%20missed%20Log%20Search%20Alerts%20and%20temporary%20unavailability%20of%20data.%3CUL%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2009%2F04%2019%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Jack%20Cantwell%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E
Version history
Last update:
‎Sep 04 2021 10:34 PM
Updated by: