%3CLINGO-SUB%20id%3D%22lingo-sub-730247%22%20slang%3D%22en-US%22%3EExperiencing%20Data%20Latency%20issue%20for%20Log%20analytics%20in%20Azure%20Portal%20-%2006%2F30%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-730247%22%20slang%3D%22en-US%22%3EFinal%20Update%3A%20Sunday%2C%2030%20June%202019%2022%3A59%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3E%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%206%2F30%2C%2010%3A05%20PM%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%206%2F30%2C12%20AM%20UTC%20and%20that%20during%20the%20~%2022%20hours%26nbsp%3B%20that%20it%20took%20to%20resolve%20the%20issue%20customers%20in%20East%20US%20experienced%20Data%20latency%20issues%20while%20accessing%20their%20data%20in%20Log%20analytics.%3CBR%20%2F%3E%20Root%20Cause%3A%20The%20failure%20was%20due%20to%20recent%20configuration%20change%20to%20a%20backend%20service%20which%20exhausted%20available%20resources.%20Incident%20Timeline%3A%26nbsp%3B%2022%20hours%2C%206%2F30%2012%20AM%20UTC%20through%206%2F30%2C10%3A05%20PM%20UTCWe%20understand%20that%20customers%20rely%20on%20Log%20analytics%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Anupama%3CBR%20%2F%3E%20Update%3A%20Sunday%2C%2030%20June%202019%2019%3A50%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3E%3CBR%20%2F%3E%3CBR%20%2F%3ERoot%20cause%20has%20been%20isolated%20to%20a%20recent%20change%20to%20a%20backend%20service%20which%20exhausted%20available%20resources.%20To%20address%20this%20issue%20engineers%20have%20reconfigured%20the%20backend%20instance%20and%20scaled%20out%20the%20resources.%20Azure%20diagnostic%20logs%20from%20Azure%20storage%20is%20now%20working%20as%20expected.%20Some%20customers%20may%20experience%20some%20latency%20and%20we%20estimate%206%20hours%20before%20all%20latent%20data%20is%20addressed.%3CBR%20%2F%3E%20Work%20Around%3ANone%20Next%20Update%3A%20Before%2007%2F01%2002%3A00%20UTC-Anupama%3CBR%20%2F%3E%20Initial%20Update%3A%20Sunday%2C%2030%20June%202019%2017%3A21%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3E%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Log%20Analytics%20and%20are%20actively%20investigating.%20Some%20customers%20of%20Log%20analytics%20in%20East%20US%20region%20may%20experience%20intermittent%20Data%20latency%20when%20accessing%20Azure%20Diagnostic%20logs%20from%20Azure%20Storage%20in%20Log%20Analytics%20workspace.%3CBR%20%2F%3E%20Work%20Around%3A%20None%20Next%20Update%3A%20Before%2006%2F30%2019%3A30%20UTCWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Anupama%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-730247%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Log%20Analytics%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Sunday, 30 June 2019 22:59 UTC

We've confirmed that all systems are back to normal with no customer impact as of 6/30, 10:05 PM UTC. Our logs show the incident started on 6/30,12 AM UTC and that during the ~ 22 hours  that it took to resolve the issue customers in East US experienced Data latency issues while accessing their data in Log analytics.
  • Root Cause: The failure was due to recent configuration change to a backend service which exhausted available resources.
  • Incident Timeline:  22 hours, 6/30 12 AM UTC through 6/30,10:05 PM UTC
We understand that customers rely on Log analytics as a critical service and apologize for any impact this incident caused.

-Anupama

Update: Sunday, 30 June 2019 19:50 UTC

Root cause has been isolated to a recent change to a backend service which exhausted available resources. To address this issue engineers have reconfigured the backend instance and scaled out the resources. Azure diagnostic logs from Azure storage is now working as expected. Some customers may experience some latency and we estimate 6 hours before all latent data is addressed.
  • Work Around:None
  • Next Update: Before 07/01 02:00 UTC
-Anupama

Initial Update: Sunday, 30 June 2019 17:21 UTC

We are aware of issues within Log Analytics and are actively investigating. Some customers of Log analytics in East US region may experience intermittent Data latency when accessing Azure Diagnostic logs from Azure Storage in Log Analytics workspace.
  • Work Around: None
  • Next Update: Before 06/30 19:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Anupama