%3CLINGO-SUB%20id%3D%22lingo-sub-2108552%22%20slang%3D%22en-US%22%3EExperiencing%20Data%20Latency%20in%20Activity%20Logs%20-%2001%2F29%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2108552%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Saturday%2C%2030%20January%202021%2000%3A45%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%201%2F30%2C%2000%3A30%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%201%2F29%2C%2002%3A00%20UTC%20and%20that%20during%20the%2022%20hours%20and%2030%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%20some%20customers%20could%20have%20experienced%20Data%20Latency%20with%20Activity%20Logs%20that%20are%20generated%20from%20Azure%20Resource%20manager%20as%20well%20as%20delayed%20Activity%20Log%20Alerts.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20a%20backend%20dependency.%3C%2FLI%3E%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%2022%20Hours%20%26amp%3B%2030%20minutes%20-%201%2F29%2C%2002%3A00%20UTC%20through%201%2F30%2C%2000%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Azure%20Monitor%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Eric%20Singleton%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Friday%2C%2029%20January%202021%2022%3A27%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3ERoot%20cause%20has%20been%20isolated%20to%20a%20misconfiguration%20from%20a%20backend%20dependency.%20To%20address%20this%20issue%20the%20team%20is%20currently%20implementing%20a%20hotfix%20to%20decrease%20the%20latency.%20Some%20customers%20may%20experience%20Data%20Latency%20with%20Activity%20Logs%20that%20are%20generated%20from%20Azure%20Resource%20Manager%20as%20well%20as%20delayed%20Activity%20Log%20Alerts.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2001%2F30%2002%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Eric%20Singleton%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Friday%2C%2029%20January%202021%2020%3A36%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Activity%20Logs%20and%20are%20actively%20investigating.%20Some%20customers%20may%20experience%20Data%20Latency%20with%20Activity%20Logs%20that%20are%20generated%20from%20Azure%20Resource%20Manager.%20Investigation%20points%20to%20a%20start%20time%20of%201%2F29%2002%3A00%20UTC.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2001%2F29%2023%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Eric%20Singleton%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2108552%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EActivity%20Log%20Alerts%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EApplication%20Insights%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%20Log%20Analytics%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Saturday, 30 January 2021 00:45 UTC

We've confirmed that all systems are back to normal with no customer impact as of 1/30, 00:30 UTC. Our logs show the incident started on 1/29, 02:00 UTC and that during the 22 hours and 30 minutes that it took to resolve the issue some customers could have experienced Data Latency with Activity Logs that are generated from Azure Resource manager as well as delayed Activity Log Alerts.
  • Root Cause: The failure was due to a backend dependency.
  • Incident Timeline: 22 Hours & 30 minutes - 1/29, 02:00 UTC through 1/30, 00:30 UTC
We understand that customers rely on Azure Monitor as a critical service and apologize for any impact this incident caused.

-Eric Singleton

Update: Friday, 29 January 2021 22:27 UTC

Root cause has been isolated to a misconfiguration from a backend dependency. To address this issue the team is currently implementing a hotfix to decrease the latency. Some customers may experience Data Latency with Activity Logs that are generated from Azure Resource Manager as well as delayed Activity Log Alerts.
  • Work Around: None
  • Next Update: Before 01/30 02:30 UTC
-Eric Singleton

Initial Update: Friday, 29 January 2021 20:36 UTC

We are aware of issues within Activity Logs and are actively investigating. Some customers may experience Data Latency with Activity Logs that are generated from Azure Resource Manager. Investigation points to a start time of 1/29 02:00 UTC.
  • Work Around: None
  • Next Update: Before 01/29 23:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Eric Singleton