Experiencing Data Latency for Log Analytics - 06/17 - Resolved

Published Jun 17 2022 04:59 AM 777 Views
Final Update: Friday, 17 June 2022 20:00 UTC

We've confirmed that all systems are back to normal with no customer impact as of 6/17, 20:00 UTC. Our logs show the incident started on 06/16, 15:30 UTC and that during the 28.02 hours that it took to resolve the issue 97% of customers experienced intermittent data latency and incorrect alert activation in Canada Central region.
  • Root Cause: A recent deployment introduced an incorrect configuration causing the data latency and incorrect alert activation to occur.
  • Incident Timeline: 28 Hours & 2 minutes - 6/16, 15:30 UTC through 6/17, 19:32 UTC
  • Mitigation: Fixed the incorrect configuration in order to mitigate the issue.
We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused.

-Shwetha

Update: Friday, 17 June 2022 11:49 UTC

Root cause has been isolated to configuration error which was impacting Log Analytics .To address this issue  reverting the configuration changes that we made. Some customers may continue to experience  intermittent data latency and incorrect alert activation.
Issue was started on 2022/06/16 15:30 UTC.
  • Work Around: None
  • Next Update: Before 06/17 18:00 UTC
-Srikanth

%3CLINGO-SUB%20id%3D%22lingo-sub-3520221%22%20slang%3D%22en-US%22%3EExperiencing%20Data%20Latency%20for%20Log%20Analytics%20-%2006%2F17%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-3520221%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Friday%2C%2017%20June%202022%2020%3A00%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%206%2F17%2C%2020%3A00%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2006%2F16%2C%2015%3A30%20UTC%20and%20that%20during%20the%2028.02%20hours%20that%20it%20took%20to%20resolve%20the%20issue%2097%25%20of%20customers%20experienced%20intermittent%20data%20latency%20and%20incorrect%20alert%20activation%20in%20Canada%20Central%20region.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20A%20recent%20deployment%20introduced%20an%20incorrect%20configuration%20causing%20the%20data%20latency%20and%20incorrect%20alert%20activation%20to%20occur.%3C%2FLI%3E%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%2028%20Hours%20%26amp%3B%202%20minutes%20-%206%2F16%2C%2015%3A30%20UTC%20through%206%2F17%2C%2019%3A32%20UTC%3C%2FLI%3E%3CLI%3E%3CU%3EMitigation%3A%3C%2FU%3E%26nbsp%3BFixed%20the%20incorrect%20configuration%20in%20order%20to%20mitigate%20the%20issue.%3C%2FLI%3E%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Azure%20Log%20Analytics%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Shwetha%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Friday%2C%2017%20June%202022%2011%3A49%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3ERoot%20cause%20has%20been%20isolated%20to%20configuration%20error%20which%20was%20impacting%20Log%20Analytics%20.To%20address%20this%20issue%26nbsp%3B%20reverting%20the%20configuration%20changes%20that%20we%20made.%20Some%20customers%20may%26nbsp%3Bcontinue%20to%20experience%26nbsp%3B%20intermittent%20data%20latency%20and%20incorrect%20alert%20activation.%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3EIssue%20was%20started%20on%202022%2F06%2F16%2015%3A30%20UTC.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2006%2F17%2018%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Srikanth%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E
Version history
Last update:
‎Jun 17 2022 01:44 PM
Updated by: