Experiencing Latency and Data Loss for Application Insights (Brazil South) - 05/09

Published May 09 2022 09:48 AM 493 Views

Final Update: Monday, 09 May 2022 16:58 UTC

We've confirmed that all systems are back to normal with no customer impact as of 5/9, 16:28 UTC. Our logs show the incident started on 5/9, 15:47 UTC and that during the 41 minutes that it took to resolve the issue customers using instrumentation-key based ingestion may have experienced intermittent data gaps and incorrect alert activation. In addition, customers using Live Metrics in Brazil South  may have experienced intermittent data gaps on Live Metrics charts.

  • Root Cause: The failure was due to cross regional connectivity issues between our services in Brazil South.

  • Incident Timeline: 41 minutes - 5/9, 15:47 UTC through 5/9, 16:28 UTC

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Ian


Initial Update: Monday, 09 May 2022 16:47 UTC


Customers with Application Insights components in Brazil South starting at 15:47 UTC may have experienced intermittent data gaps and incorrect alert activation.

  • Work Around: none

  • Next Update: Before 05/09 19:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-Ian


%3CLINGO-SUB%20id%3D%22lingo-sub-3345266%22%20slang%3D%22en-US%22%3EExperiencing%20Latency%20and%20Data%20Loss%20for%20Application%20Insights%20(Brazil%20South)%20-%2005%2F09%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-3345266%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CP%20class%3D%22MsoNormal%22%20style%3D%22margin-bottom%3A%200in%3B%20line-height%3A%20normal%3B%20background-image%3A%20initial%3B%20background-position%3A%20initial%3B%20background-size%3A%20initial%3B%20background-repeat%3A%20initial%3B%20background-attachment%3A%20initial%3B%20background-origin%3A%20initial%3B%20background-clip%3A%20initial%3B%22%3E%3CU%3E%3CSPAN%20style%3D%22font-size%3A%2010.5pt%3B%20font-family%3A%20Roboto%3B%22%3EFinal%20Update%3C%2FSPAN%3E%3C%2FU%3E%3CSPAN%20style%3D%22font-size%3A%2010.5pt%3B%20font-family%3A%20Roboto%3B%22%3E%3A%20Monday%2C%2009%20May%202022%2016%3A58%20UTC%3CBR%20%2F%3E%20%3CBR%20%2F%3E%20We've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%205%2F9%2C%2016%3A28%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%205%2F9%2C%2015%3A47%20UTC%20and%20that%20during%20the%2041%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%20customers%20using%20instrumentation-key%20based%20ingestion%20may%20have%20experienced%20intermittent%20data%20gaps%20and%20incorrect%20alert%20activation.%20In%20addition%2C%20customers%20using%20Live%20Metrics%20in%20Brazil%20South%26nbsp%3B%20may%20have%20experienced%20intermittent%20data%20gaps%20on%20Live%20Metrics%20charts.%20%3CBR%20%2F%3E%20%3C!--%5Bif%20!supportLineBreakNewLine%5D--%3E%3CBR%20%2F%3E%20%3C!--%5Bendif%5D--%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3C%2FP%3E%3CP%3E%3C%2FP%3E%0A%0A%3CUL%20type%3D%22disc%22%3E%0A%20%3CLI%20class%3D%22MsoNormal%22%20style%3D%22line-height%3A%20normal%3B%20background-image%3A%20initial%3B%20background-position%3A%20initial%3B%20background-size%3A%20initial%3B%20background-repeat%3A%20initial%3B%20background-attachment%3A%20initial%3B%20background-origin%3A%20initial%3B%20background-clip%3A%20initial%3B%22%3E%3CU%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bfont-family%3ARoboto%3B%0A%20%20%20%20%20mso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bmso-bidi-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3B%0A%20%20%20%20%20mso-font-kerning%3A0pt%3Bmso-ligatures%3Anone%22%3ERoot%20Cause%3C%2FSPAN%3E%3C%2FU%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bfont-family%3ARoboto%3Bmso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3B%0A%20%20%20%20%20mso-bidi-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bmso-font-kerning%3A0pt%3Bmso-ligatures%3A%0A%20%20%20%20%20none%22%3E%3A%20The%20failure%20was%20due%20to%20cross%20regional%20connectivity%20issues%20between%20our%20services%20in%20Brazil%20South.%3CP%3E%3C%2FP%3E%3C%2FSPAN%3E%3C%2FLI%3E%0A%20%3CLI%20class%3D%22MsoNormal%22%20style%3D%22line-height%3A%20normal%3B%20background-image%3A%20initial%3B%20background-position%3A%20initial%3B%20background-size%3A%20initial%3B%20background-repeat%3A%20initial%3B%20background-attachment%3A%20initial%3B%20background-origin%3A%20initial%3B%20background-clip%3A%20initial%3B%22%3E%3CU%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bfont-family%3ARoboto%3B%0A%20%20%20%20%20mso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bmso-bidi-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3B%0A%20%20%20%20%20mso-font-kerning%3A0pt%3Bmso-ligatures%3Anone%22%3EIncident%20Timeline%3C%2FSPAN%3E%3C%2FU%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bfont-family%3ARoboto%3Bmso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3B%0A%20%20%20%20%20mso-bidi-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bmso-font-kerning%3A0pt%3Bmso-ligatures%3A%0A%20%20%20%20%20none%22%3E%3A%2041%20minutes%20-%205%2F9%2C%2015%3A47%20UTC%20through%205%2F9%2C%2016%3A28%20UTC%3CP%3E%3C%2FP%3E%3C%2FSPAN%3E%3C%2FLI%3E%0A%3C%2FUL%3E%0A%0A%3CP%20class%3D%22MsoNormal%22%20style%3D%22margin-bottom%3A%200in%3B%20line-height%3A%20normal%3B%20background-image%3A%20initial%3B%20background-position%3A%20initial%3B%20background-size%3A%20initial%3B%20background-repeat%3A%20initial%3B%20background-attachment%3A%20initial%3B%20background-origin%3A%20initial%3B%20background-clip%3A%20initial%3B%22%3E%3CSPAN%20style%3D%22font-size%3A%2010.5pt%3B%20font-family%3A%20Roboto%3B%22%3EWe%20understand%20that%20customers%20rely%20on%20Application%20Insights%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%20%3CBR%20%2F%3E%20-Ian%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3C%2FP%3E%3CP%3E%3C%2FP%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CP%20class%3D%22MsoNormal%22%20style%3D%22margin-bottom%3A%200in%3B%20line-height%3A%20normal%3B%20background-image%3A%20initial%3B%20background-position%3A%20initial%3B%20background-size%3A%20initial%3B%20background-repeat%3A%20initial%3B%20background-attachment%3A%20initial%3B%20background-origin%3A%20initial%3B%20background-clip%3A%20initial%3B%22%3E%3CU%3E%3CSPAN%20style%3D%22font-size%3A%2010.5pt%3B%20font-family%3A%20Roboto%3B%22%3EInitial%20Update%3C%2FSPAN%3E%3C%2FU%3E%3CSPAN%20style%3D%22font-size%3A%2010.5pt%3B%20font-family%3A%20Roboto%3B%22%3E%3A%20Monday%2C%2009%20May%202022%2016%3A47%20UTC%3CBR%20%2F%3E%20%3CBR%20%2F%3E%20%3C!--%5Bif%20!supportLineBreakNewLine%5D--%3E%3CBR%20%2F%3E%20%3C!--%5Bendif%5D--%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3C%2FP%3E%3CP%3E%3C%2FP%3E%0A%0A%3CP%20class%3D%22MsoNormal%22%20style%3D%22margin-bottom%3A%200in%3B%20line-height%3A%20normal%3B%20background-image%3A%20initial%3B%20background-position%3A%20initial%3B%20background-size%3A%20initial%3B%20background-repeat%3A%20initial%3B%20background-attachment%3A%20initial%3B%20background-origin%3A%20initial%3B%20background-clip%3A%20initial%3B%22%3E%3CSPAN%20style%3D%22font-size%3A%2010.5pt%3B%20font-family%3A%20Roboto%3B%22%3ECustomers%20with%20Application%20Insights%20components%20in%20Brazil%20South%20starting%20at%2015%3A47%20UTC%20may%20have%20experienced%20intermittent%20data%20gaps%20and%20incorrect%20alert%20activation.%3CBR%20%2F%3E%3CBR%20%2F%3E%20%3C!--%5Bendif%5D--%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3C%2FP%3E%3CP%3E%3C%2FP%3E%0A%0A%3CUL%20type%3D%22disc%22%3E%0A%20%3CLI%20class%3D%22MsoNormal%22%20style%3D%22line-height%3A%20normal%3B%20background-image%3A%20initial%3B%20background-position%3A%20initial%3B%20background-size%3A%20initial%3B%20background-repeat%3A%20initial%3B%20background-attachment%3A%20initial%3B%20background-origin%3A%20initial%3B%20background-clip%3A%20initial%3B%22%3E%3CU%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bfont-family%3ARoboto%3B%0A%20%20%20%20%20mso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bmso-bidi-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3B%0A%20%20%20%20%20mso-font-kerning%3A0pt%3Bmso-ligatures%3Anone%22%3EWork%20Around%3C%2FSPAN%3E%3C%2FU%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bfont-family%3ARoboto%3Bmso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3B%0A%20%20%20%20%20mso-bidi-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bmso-font-kerning%3A0pt%3Bmso-ligatures%3A%0A%20%20%20%20%20none%22%3E%3A%20none%3CP%3E%3C%2FP%3E%3C%2FSPAN%3E%3C%2FLI%3E%0A%20%3CLI%20class%3D%22MsoNormal%22%20style%3D%22line-height%3A%20normal%3B%20background-image%3A%20initial%3B%20background-position%3A%20initial%3B%20background-size%3A%20initial%3B%20background-repeat%3A%20initial%3B%20background-attachment%3A%20initial%3B%20background-origin%3A%20initial%3B%20background-clip%3A%20initial%3B%22%3E%3CU%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bfont-family%3ARoboto%3B%0A%20%20%20%20%20mso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bmso-bidi-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3B%0A%20%20%20%20%20mso-font-kerning%3A0pt%3Bmso-ligatures%3Anone%22%3ENext%20Update%3C%2FSPAN%3E%3C%2FU%3E%3CSPAN%20style%3D%22font-size%3A10.5pt%3Bfont-family%3ARoboto%3Bmso-fareast-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3B%0A%20%20%20%20%20mso-bidi-font-family%3A%26quot%3BTimes%20New%20Roman%26quot%3B%3Bmso-font-kerning%3A0pt%3Bmso-ligatures%3A%0A%20%20%20%20%20none%22%3E%3A%20Before%2005%2F09%2019%3A00%20UTC%3CP%3E%3C%2FP%3E%3C%2FSPAN%3E%3C%2FLI%3E%0A%3C%2FUL%3E%0A%0A%3CP%20class%3D%22MsoNormal%22%20style%3D%22margin-bottom%3A%200in%3B%20line-height%3A%20normal%3B%20background-image%3A%20initial%3B%20background-position%3A%20initial%3B%20background-size%3A%20initial%3B%20background-repeat%3A%20initial%3B%20background-attachment%3A%20initial%3B%20background-origin%3A%20initial%3B%20background-clip%3A%20initial%3B%22%3E%3CSPAN%20style%3D%22font-size%3A%2010.5pt%3B%20font-family%3A%20Roboto%3B%22%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E%20-Ian%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3C%2FP%3E%3CP%3E%3C%2FP%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E
Version history
Last update:
‎May 09 2022 10:09 AM
Updated by: