%3CLINGO-SUB%20id%3D%22lingo-sub-2160526%22%20slang%3D%22en-US%22%3EExperiencing%20Data%20Access%20Issue%20in%20Azure%20portal%20for%20Service%20Map%20-%2002%2F23%20-%20Investigating%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2160526%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Wednesday%2C%2024%20February%202021%2001%3A32%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20continue%20to%20investigate%20issues%20within%20Service%20Map.%20Root%20cause%20is%20due%20to%20one%20of%20the%20backend%20component%20becoming%20inaccessible%20after%20a%20scheduled%20drill%2Ftest%20in%26nbsp%3B%3CSPAN%20style%3D%22font-family%3A%20Calibri%2C%20sans-serif%3B%20font-size%3A%2014.6667px%3B%22%3EEastUS2EUAP%3C%2FSPAN%3E%3CSPAN%20style%3D%22font-family%3A%20Calibri%2C%20sans-serif%3B%20font-size%3A%2014.6667px%3B%22%3E%26nbsp%3B%20region%3C%2FSPAN%3E.%3CSPAN%20style%3D%22font-family%3A%20Calibri%2C%20sans-serif%3B%20font-size%3A%2011pt%3B%22%3E%26nbsp%3B%3C%2FSPAN%3ESome%20customers%20who%20have%20VM's%20sending%20data%20to%26nbsp%3B%3CSPAN%20style%3D%22font-family%3A%20Calibri%2C%20sans-serif%3B%20font-size%3A%2014.6667px%3B%22%3EEastUS2EUAP%3C%2FSPAN%3E%3CSPAN%20style%3D%22font-family%3A%20Calibri%2C%20sans-serif%3B%20font-size%3A%2014.6667px%3B%22%3E%26nbsp%3B%20workspaces%2C%26nbsp%3B%3C%2FSPAN%3Econtinue%20to%20experience%20issues%20seeing%20performance%20data%20in%20Azure%20monitor%20when%20Azure%20mode%20is%20selected.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2002%2F24%2014%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Anupama%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Tuesday%2C%2023%20February%202021%2021%3A22%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20continue%20to%20investigate%20issues%20within%20Service%20map.%20Root%20cause%20is%20due%20to%20one%20of%20the%20backend%20component%20becoming%20inaccessible%20after%20a%20scheduled%20drill%2Ftest%20in%26nbsp%3B%3CSPAN%20style%3D%22font-family%3A%20Calibri%2C%20sans-serif%3B%20font-size%3A%2014.6667px%3B%22%3EEastUS2EUAP%3C%2FSPAN%3E%3CSPAN%20style%3D%22font-family%3A%20Calibri%2C%20sans-serif%3B%20font-size%3A%2014.6667px%3B%22%3E%26nbsp%3B%20region%3C%2FSPAN%3E.%3CSPAN%20style%3D%22font-family%3A%20Calibri%2C%20sans-serif%3B%20font-size%3A%2011pt%3B%22%3E%26nbsp%3B%3C%2FSPAN%3ESome%20customers%20who%20have%20VM's%20sending%20data%20to%26nbsp%3B%3CSPAN%20style%3D%22font-family%3A%20Calibri%2C%20sans-serif%3B%20font-size%3A%2014.6667px%3B%22%3EEastUS2EUAP%3C%2FSPAN%3E%3CSPAN%20style%3D%22font-family%3A%20Calibri%2C%20sans-serif%3B%20font-size%3A%2014.6667px%3B%22%3E%26nbsp%3B%20workspaces%2C%26nbsp%3B%3C%2FSPAN%3Econtinue%20to%20experience%20issues%20seeing%20performance%20data%20in%20Azure%20monitor%20when%20Azure%20mode%20is%20selected.%20The%20issue%20started%20at%2016%3A00%20UTC%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2002%2F23%2023%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3E-Anupama%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2160526%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EService%20Map%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Thursday, 25 February 2021 19:55 UTC

We've confirmed that all systems are back to normal with no customer impact as of 2/25, 18:40 UTC. Our logs show the incident started on 2/15 and that during the duration of  ~10 days that it took to resolve the issue some customers experienced issues seeing performance data in Azure monitor when Azure mode is selected and 5xx errors while viewing performance data on browser.
  • Root Cause: The failure was due to one of the back end services turning unhealthy after a deployment.
  • Incident Timeline:  2/15 through 2/25, 18:40 UTC
We understand that customers rely on Azure monitor for VM's as a critical service and apologize for any impact this incident caused.

-Anupama

Update: Thursday, 25 February 2021 04:22 UTC

Root cause has been isolated to service turning unhealthy after incorrect deployment which was impacting performance data view in Azure monitor when Azure mode is selected. To address this issue engineers are rolling back the service to previous healthy state. Some customers who have VM's sending data to workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected and 5xx errors while viewing performance data on browser.
  • Work Around: None
  • Next Update: Before 02/26 04:30 UTC
-Anupama

Update: Thursday, 25 February 2021 01:28 UTC

Root cause has been isolated to service turning unhealthy after incorrect deployment which was impacting performance data view in Azure monitor when Azure mode is selected. To address this issue engineers are rolling back the service to previous healthy state. Some customers who have VM's sending data to workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected and 5xx errors while viewing performance data on browser.
  • Work Around: None
  • Next Update: Before 02/25 05:30 UTC
-Anupama

Update: Thursday, 25 February 2021 00:01 UTC

Root cause has been isolated to service turning unhealthy after incorrect deployment which was impacting performance data view in Azure monitor when Azure mode is selected. To address this issue we are rolling out a fix to the service. Some customers who have VM's sending data to workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected and 5xx errors while viewing performance data on browser.
  • Work Around: None
  • Next Update: Before 02/25 02:30 UTC
-Anupama

Update: Wednesday, 24 February 2021 22:01 UTC

We continue to investigate issues within Azure monitor for VM's. Preliminary investigation showed that the root cause was due to one of the back end component becoming inaccessible after a scheduled drill/test in EastUS2EUAP region. Further investigation shows that the root cause is due to regression that happened after a deployment on one of the back end services. Some customers who have VM's sending data to workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected. Customers might also see 5xx errors while viewing the performance data on the portal.

  • Work Around: None
  • Next Update: Before 02/25 00:30 UTC
-Anupama

Update: Wednesday, 24 February 2021 12:02 UTC

We continue to investigate issues within Service Map. Root cause is due to one of the backend component becoming inaccessible after a scheduled drill/test in EastUS2EUAP region. Some customers who have VM's sending data to EastUS2EUAP workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected.
  • Work Around: None
  • Next Update: Before 02/25 00:30 UTC
-Harshita

Update: Wednesday, 24 February 2021 01:32 UTC

We continue to investigate issues within Service Map. Root cause is due to one of the backend component becoming inaccessible after a scheduled drill/test in EastUS2EUAP  region. Some customers who have VM's sending data to EastUS2EUAP  workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected.
  • Work Around: None
  • Next Update: Before 02/24 14:00 UTC
-Anupama

Update: Tuesday, 23 February 2021 21:22 UTC

We continue to investigate issues within Service map. Root cause is due to one of the backend component becoming inaccessible after a scheduled drill/test in EastUS2EUAP  region. Some customers who have VM's sending data to EastUS2EUAP  workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected. The issue started at 16:00 UTC
  • Work Around: None
  • Next Update: Before 02/23 23:30 UTC
-Anupama