%3CLINGO-SUB%20id%3D%22lingo-sub-1468107%22%20slang%3D%22en-US%22%3EExperiencing%20Data%20Latency%20Issues%20in%20Log%20Analytics%20for%20Virginia%20region%20-%2006%2F16%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1468107%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Tuesday%2C%2016%20June%202020%2020%3A01%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3E%3CSPAN%20style%3D%22font-family%3A%20Helvetica%3B%22%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2006%2F16%2C%2018%3A16%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2006%2F15%2C%2020%3A32%20UTC%20and%20that%20during%20the%2019%20hours%2044%20min%20that%20it%20took%20to%20resolve%20the%20issue%26nbsp%3B%3CSPAN%20style%3D%22font-size%3A%2014.6667px%3B%22%3Ecustomer%20using%20Log%20Analytics%20in%20US%20Gov%20Virginia%20who%20may%20have%20experienced%20log%20processing%20delays%20with%20Network%20Performance%20Monitor%20Solution%20messages%20and%20missing%20alerts%20in%20this%20region%3C%2FSPAN%3E%3C%2FSPAN%3E.%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CUL%3E%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20We%20determined%20that%20a%20certificate%20rotated%20from%20a%20recent%20deployment%20made%20Network%20Performance%20Monitor%20Solution%20messages%20unable%20to%20ingest%20properly.%3CBR%20%2F%3E%3C%2FLI%3E%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%2019%20Hours%20%26amp%3B%2044%20minutes%20-%2006%2F15%2C%2020%3A32%20UTC%20through%2006%2F16%2C%2018%3A16%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Azure%20Log%20Analytics%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Subhash%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Tuesday%2C%2016%20June%202020%2018%3A13%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3E%3CUL%3E%3CDIV%20style%3D%22%22%3E%3CSPAN%20style%3D%22font-family%3A%20Helvetica%3B%22%3EWe%20continue%20to%20investigate%20issues%20within%20Log%20Analytics.%20Root%20cause%20is%20not%20fully%20understood%20at%20this%20time.%20Some%20customers%20in%20Fairfax%20region%20may%20experience%20log%20processing%20delays%20of%20NPM%20messages%26nbsp%3Bas%20well%20as%20misfiring%20alerts%20based%20on%20these%20messages%26nbsp%3Bin%20this%26nbsp%3Bregions%26nbsp%3B.%20We%20are%20working%20to%20establish%20the%20start%20time%20for%20the%20issue%2C%20initial%20findings%20indicate%20that%20the%20problem%20began%20at%2006%2F15%2020%3A32%20UTC.%20Resolution%20of%20ingestion%20for%201%20hour.%20resolution.%3C%2FSPAN%3E%3C%2FDIV%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2006%2F16%2022%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Subhash%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Tuesday%2C%2016%20June%202020%2015%3A31%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Log%20Analytics%20in%20Virginia%20region%20in%20US%20Gov.%20Some%20customers%20may%20experience%20log%20processing%20delays%20of%20NPM%20messages.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2006%2F16%2018%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Santhosh%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3A%20lightgray%3B%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1468107%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Log%20Analytics%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Tuesday, 16 June 2020 20:01 UTC

We've confirmed that all systems are back to normal with no customer impact as of 06/16, 18:16 UTC. Our logs show the incident started on 06/15, 20:32 UTC and that during the 19 hours 44 min that it took to resolve the issue customer using Log Analytics in US Gov Virginia who may have experienced log processing delays with Network Performance Monitor Solution messages and missing alerts in this region.
  • Root Cause: We determined that a certificate rotated from a recent deployment made Network Performance Monitor Solution messages unable to ingest properly.
  • Incident Timeline: 19 Hours & 44 minutes - 06/15, 20:32 UTC through 06/16, 18:16 UTC
We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused.

-Subhash

Update: Tuesday, 16 June 2020 18:13 UTC

    We continue to investigate issues within Log Analytics. Root cause is not fully understood at this time. Some customers in Fairfax region may experience log processing delays of NPM messages as well as misfiring alerts based on these messages in this regions . We are working to establish the start time for the issue, initial findings indicate that the problem began at 06/15 20:32 UTC. Resolution of ingestion for 1 hour. resolution.
  • Work Around: None
  • Next Update: Before 06/16 22:30 UTC
-Subhash

Initial Update: Tuesday, 16 June 2020 15:31 UTC

We are aware of issues within Log Analytics in Virginia region in US Gov. Some customers may experience log processing delays of NPM messages.
  • Work Around: None
  • Next Update: Before 06/16 18:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Santhosh