Experiencing Data Ingestion Issue for Log Analytics - 02/18 - Resolved

Published Feb 18 2021 10:57 AM 961 Views
Final Update: Thursday, 18 February 2021 18:55 UTC

We've confirmed that all systems are back to normal with no customer impact as of 2/18, 16:50 UTC. Our logs show the incident started on 2/18, 13:10 UTC and that during the 3 hours 40 min that it took to resolve the issue, customers in West Europe and West Central US using Log Analytics may have not seen heartbeat events through the Log Analytics workspace and also would have experienced incorrect alerting on heartbeat events. Additionally, between 19:35 UTC on 17 Feb 2021 to 16:50 UTC on 18 Feb 2021, a subset of customers were not able to see the following columns: SubscriptionId, Resource, ResourceId, ResourceType through the Azure Portal, Azure CLI or Power Shell.
  • Root Cause: The failure was due to implementation of a new transform as part of a recent deployment. As the current deployment started, backend workflows were mistakenly deleted and that the deletion caused the alert objects to not be processed through Log Analytics.
  • Incident Timeline: 3 Hours & 40 minutes - 2/18, 13:10 UTC through 2/18, 16:50 UTC
We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused.

-Anupama

%3CLINGO-SUB%20id%3D%22lingo-sub-2147051%22%20slang%3D%22en-US%22%3EExperiencing%20Data%20Ingestion%20Issue%20for%20Log%20Analytics%20-%2002%2F18%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2147051%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Thursday%2C%2018%20February%202021%2018%3A55%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%202%2F18%2C%2016%3A50%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%202%2F18%2C%2013%3A10%20UTC%20and%20that%20during%20the%203%20hours%2040%20min%20that%20it%20took%20to%20resolve%20the%20issue%2C%20customers%20in%20West%20Europe%20and%20West%20Central%20US%20using%20Log%20Analytics%20may%20have%20not%20seen%20heartbeat%20events%20through%20the%20Log%20Analytics%20workspace%20and%20also%20would%20have%20experienced%20incorrect%20alerting%20on%20heartbeat%20events.%20Additionally%2C%20between%2019%3A35%20UTC%20on%2017%20Feb%202021%20to%2016%3A50%20UTC%20on%2018%20Feb%202021%2C%20a%20subset%20of%20customers%20were%20not%20able%20to%20see%20the%20following%20columns%3A%20SubscriptionId%2C%20Resource%2C%20ResourceId%2C%20ResourceType%20through%20the%20Azure%20Portal%2C%20Azure%20CLI%20or%20Power%20Shell.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20implementation%20of%20a%20new%20transform%20as%20part%20of%20a%20recent%20deployment.%20As%20the%20current%20deployment%20started%2C%20backend%20workflows%20were%20mistakenly%20deleted%20and%20that%20the%20deletion%20caused%20the%20alert%20objects%20to%20not%20be%20processed%20through%20Log%20Analytics.%3C%2FLI%3E%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%203%20Hours%20%26amp%3B%2040%20minutes%20-%202%2F18%2C%2013%3A10%20UTC%20through%202%2F18%2C%2016%3A50%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Azure%20Log%20Analytics%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Anupama%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2147051%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Log%20Analytics%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Version history
Last update:
‎Feb 18 2021 10:57 AM
Updated by: