Experiencing Data Latency in Europe regions for platform logs - 08/17 - Resolved

Published Aug 17 2021 10:16 AM 1,101 Views
Final Update: Wednesday, 18 August 2021 18:03 UTC

We've confirmed that all systems are back to normal as of 08/18, 17:30 UTC. Our logs show the incident started on 08/17, 09:20 UTC.  During the timeline of the incident customers experienced latency for platform logs data .
  • Root Cause: The failure was due to underlying cache infrastructure that became unhealthy.
  • Incident Timeline: 08/17, 09:20 through 08/18, 17:30 UTC.
We understand that customers rely on Azure Monitor as a critical service and apologize for any impact this incident caused.

-Chandar

Update: Wednesday, 18 August 2021 16:59 UTC

We continue to clear the backlog of log requests that have been pending. Based on progress observed so far and the trend, we are expecting the backlog to clear within the next 3 hours
  • Work Around:
  • Next Update: Before 08/18 20:00 UTC
-Chandar

Update: Wednesday, 18 August 2021 04:12 UTC

We have identified a potential root cause of an unhealthy, dependent Redis Cache and have applied mitigation to return that cache to a healthy state. We continue to clear the backlog of log requests that have been pending. Executing the safest and most effective strategy to clear the backlog could potentially take up to 10 hours.
  • Work Around: None
  • Next Update: Before 08/18 16:30 UTC
-Mohini

Update: Tuesday, 17 August 2021 22:36 UTC
We have applied mitigation steps and are working clear the backlog of log requests that have been pending. We are continuing to apply further mitigation steps in order to expedite the processing of the backlog and have engaged additional teams to assist in leveraging their resources to help quickly process pending requests. The next update will be provided in 2 hours, or as events warrant.

  • Work Around: None
  • Next Update: Before 08/18 01:00 UTC
-Chandar

Update: Tuesday, 17 August 2021 18:29 UTC

We continue to investigate issues within Azure Monitor. Root cause is not fully understood at this time. Some customers in impacted regions continue to experience  high latency for platform logs configured via Diagnostics settings.. We currently have no estimate for resolution.
  • Work Around: None
  • Next Update: Before 08/17 20:30 UTC
-Chandar

Initial Update: Tuesday, 17 August 2021 17:09 UTC

Starting at 09:20 UTC on 17 Aug 2021 you have been identified as a customer using Azure Monitor who may be experiencing high latency for platform logs configured via Diagnostics settings.
  • Work Around: None
  • Next Update: Before 08/17 18:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Chandar

%3CLINGO-SUB%20id%3D%22lingo-sub-2657065%22%20slang%3D%22en-US%22%3EExperiencing%20Data%20Latency%20in%20Europe%20regions%20for%20platform%20logs%20-%2008%2F17%20-%20Mitigating%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2657065%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Wednesday%2C%2018%20August%202021%2016%3A59%20UTC%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CBR%20%2F%3EWe%20continue%20to%20clear%20the%20backlog%20of%20log%20requests%20that%20have%20been%20pending.%20Based%20on%20progress%20observed%20so%20far%20and%20the%20trend%2C%20we%20are%20expecting%20the%20backlog%20to%20clear%20within%20the%20next%203%20hours%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20%3CNONE%20or%3D%22%22%20details%3D%22%22%3E%3C%2FNONE%3E%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2008%2F18%2020%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Chandar%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Wednesday%2C%2018%20August%202021%2004%3A12%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20have%20identified%20a%20potential%20root%20cause%20of%20an%20unhealthy%2C%20dependent%20Redis%20Cache%20and%20have%20applied%20mitigation%20to%20return%20that%20cache%20to%20a%20healthy%20state.%20We%20continue%20to%20clear%20the%20backlog%20of%20log%20requests%20that%20have%20been%20pending.%20Executing%20the%20safest%20and%20most%20effective%20strategy%20to%20clear%20the%20backlog%20could%20potentially%20take%20up%20to%2010%20hours.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2008%2F18%2016%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Mohini%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Tuesday%2C%2017%20August%202021%2022%3A36%20UTC%3CBR%20%2F%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CSPAN%20style%3D%22font-size%3A%2010pt%3B%20font-family%3A%20Helvetica%2C%20sans-serif%3B%22%3EWe%20have%20applied%20mitigation%20steps%20and%20are%20working%20clear%20the%20backlog%20of%20log%20requests%20that%20have%20been%20pending.%20We%20are%20continuing%20to%20apply%20further%20mitigation%20steps%20in%20order%20to%20expedite%20the%20processing%20of%20the%20backlog%20and%20have%20engaged%20additional%20teams%20to%20assist%20in%20leveraging%20their%20resources%20to%20help%20quickly%20process%20pending%20requests.%20The%20next%20update%20will%20be%20provided%20in%202%20hours%2C%20or%20as%20events%20warrant.%3C%2FSPAN%3E%3CBR%20%2F%3E%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2008%2F18%2001%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Chandar%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Tuesday%2C%2017%20August%202021%2018%3A29%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20continue%20to%20investigate%20issues%20within%20Azure%20Monitor.%20Root%20cause%20is%20not%20fully%20understood%20at%20this%20time.%20Some%20customers%20in%20impacted%20regions%20continue%20to%20experience%26nbsp%3B%26nbsp%3Bhigh%20latency%20for%20platform%20logs%20configured%20via%20Diagnostics%20settings..%20We%20currently%20have%20no%20estimate%20for%20resolution.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2008%2F17%2020%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Chandar%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Tuesday%2C%2017%20August%202021%2017%3A09%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EStarting%20at%2009%3A20%20UTC%20on%2017%20Aug%202021%20you%20have%20been%20identified%20as%20a%20customer%20using%20Azure%20Monitor%20who%20may%20be%20experiencing%20high%20latency%20for%20platform%20logs%20configured%20via%20Diagnostics%20settings.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2008%2F17%2018%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Chandar%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E
Version history
Last update:
‎Aug 18 2021 11:13 AM
Updated by: