Data Latency Issue and Missed or Delayed Alerts in SouthEast Asia region

Published Apr 13 2022 10:38 AM 1,036 Views

Summary of Impact: Between 15:30 UTC and 21:00 UTC on 13 Apr 2022, customers using Azure Log Analytics in Southeast Asia who may have experienced latency while querying data, delayed or missed alerts.


Preliminary root cause: We have identified that a high CPU usage allowed for an increase in the load of incoming calls to the service to cause some instances of a backend service Azure Log Analytics relies on for control plane operations to become unhealthy. This resulted in the latency and missed alerts customers may have experienced.


Mitigation: We introduced additional capacity to the region, allowing provisioning of new clusters and/or job execution to be performed in this region.

%3CLINGO-SUB%20id%3D%22lingo-sub-3284927%22%20slang%3D%22en-US%22%3EData%20Latency%20Issue%20and%20Missed%20or%20Delayed%20Alerts%20in%20SouthEast%20Asia%20region%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-3284927%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CP%20style%3D%22-webkit-font-smoothing%3A%20antialiased%3B%20margin-right%3A%200px%3B%20margin-bottom%3A%200px%3B%20margin-left%3A%200px%3B%20color%3A%20rgb(36%2C%2036%2C%2036)%3B%20font-family%3A%20-apple-system%2C%20BlinkMacSystemFont%2C%20%26quot%3BSegoe%20UI%26quot%3B%2C%20%26quot%3BApple%20Color%20Emoji%26quot%3B%2C%20%26quot%3BSegoe%20UI%20Emoji%26quot%3B%2C%20%26quot%3BSegoe%20UI%20Web%26quot%3B%2C%20sans-serif%3B%20font-size%3A%2014px%3B%20font-style%3A%20inherit%3B%20font-weight%3A%20inherit%3B%22%3ESummary%20of%20Impact%3A%20Between%2015%3A30%20UTC%20and%2021%3A00%20UTC%20on%2013%20Apr%202022%2C%20customers%20using%20Azure%20Log%20Analytics%20in%20Southeast%20Asia%20who%20may%20have%20experienced%20latency%20while%20querying%20data%2C%20delayed%20or%20missed%20alerts.%3C%2FP%3E%3CP%20style%3D%22-webkit-font-smoothing%3A%20antialiased%3B%20margin-right%3A%200px%3B%20margin-bottom%3A%200px%3B%20margin-left%3A%200px%3B%20color%3A%20rgb(36%2C%2036%2C%2036)%3B%20font-family%3A%20-apple-system%2C%20BlinkMacSystemFont%2C%20%26quot%3BSegoe%20UI%26quot%3B%2C%20%26quot%3BApple%20Color%20Emoji%26quot%3B%2C%20%26quot%3BSegoe%20UI%20Emoji%26quot%3B%2C%20%26quot%3BSegoe%20UI%20Web%26quot%3B%2C%20sans-serif%3B%20font-size%3A%2014px%3B%22%3E%3CBR%20style%3D%22-webkit-font-smoothing%3A%20antialiased%3B%22%20%2F%3E%3C%2FP%3E%3CP%20style%3D%22-webkit-font-smoothing%3A%20antialiased%3B%20margin-right%3A%200px%3B%20margin-bottom%3A%200px%3B%20margin-left%3A%200px%3B%20color%3A%20rgb(36%2C%2036%2C%2036)%3B%20font-family%3A%20-apple-system%2C%20BlinkMacSystemFont%2C%20%26quot%3BSegoe%20UI%26quot%3B%2C%20%26quot%3BApple%20Color%20Emoji%26quot%3B%2C%20%26quot%3BSegoe%20UI%20Emoji%26quot%3B%2C%20%26quot%3BSegoe%20UI%20Web%26quot%3B%2C%20sans-serif%3B%20font-size%3A%2014px%3B%22%3E%3CSPAN%20style%3D%22-webkit-font-smoothing%3A%20antialiased%3B%20font-style%3A%20inherit%3B%20font-weight%3A%20inherit%3B%22%3EPreliminary%20root%20cause%3A%20We%20have%20identified%20that%20a%20high%20CPU%20usage%20allowed%20for%20an%20increase%20in%20the%20load%20of%20incoming%20calls%20to%20the%20service%20to%20cause%20some%20instances%20of%20a%20backend%20service%20Azure%20Log%20Analytics%20relies%20on%20for%20control%20plane%20operations%20to%20become%20unhealthy.%20This%20resulted%20in%20the%20latency%20and%20missed%20alerts%20customers%20may%20have%20experienced.%3C%2FSPAN%3E%3CBR%20style%3D%22-webkit-font-smoothing%3A%20antialiased%3B%22%20%2F%3E%3C%2FP%3E%3CP%20style%3D%22-webkit-font-smoothing%3A%20antialiased%3B%20margin-right%3A%200px%3B%20margin-bottom%3A%200px%3B%20margin-left%3A%200px%3B%20color%3A%20rgb(36%2C%2036%2C%2036)%3B%20font-family%3A%20-apple-system%2C%20BlinkMacSystemFont%2C%20%26quot%3BSegoe%20UI%26quot%3B%2C%20%26quot%3BApple%20Color%20Emoji%26quot%3B%2C%20%26quot%3BSegoe%20UI%20Emoji%26quot%3B%2C%20%26quot%3BSegoe%20UI%20Web%26quot%3B%2C%20sans-serif%3B%20font-size%3A%2014px%3B%22%3E%3CBR%20style%3D%22-webkit-font-smoothing%3A%20antialiased%3B%22%20%2F%3E%3C%2FP%3E%3CP%20style%3D%22-webkit-font-smoothing%3A%20antialiased%3B%20margin-right%3A%200px%3B%20margin-bottom%3A%200px%3B%20margin-left%3A%200px%3B%20color%3A%20rgb(36%2C%2036%2C%2036)%3B%20font-family%3A%20-apple-system%2C%20BlinkMacSystemFont%2C%20%26quot%3BSegoe%20UI%26quot%3B%2C%20%26quot%3BApple%20Color%20Emoji%26quot%3B%2C%20%26quot%3BSegoe%20UI%20Emoji%26quot%3B%2C%20%26quot%3BSegoe%20UI%20Web%26quot%3B%2C%20sans-serif%3B%20font-size%3A%2014px%3B%22%3E%3CSPAN%20style%3D%22-webkit-font-smoothing%3A%20antialiased%3B%20font-style%3A%20inherit%3B%20font-weight%3A%20inherit%3B%22%3EMitigation%3A%20We%20introduced%20additional%20capacity%20to%20the%20region%2C%20allowing%20provisioning%20of%20new%20clusters%20and%2For%20job%20execution%20to%20be%20performed%20in%20this%20region.%3C%2FSPAN%3E%3C%2FP%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E
Version history
Last update:
‎Apr 13 2022 04:20 PM
Updated by: