Final Update: Thursday, 25 February 2021 19:55 UTC
We've confirmed that all systems are back to normal with no customer impact as of 2/25, 18:40 UTC. Our logs show the incident started on 2/15 and that during the duration of ~10 days that it took to resolve the issue some customers experienced issues seeing performance data in Azure monitor when Azure mode is selected and 5xx errors while viewing performance data on browser.
-Anupama
We've confirmed that all systems are back to normal with no customer impact as of 2/25, 18:40 UTC. Our logs show the incident started on 2/15 and that during the duration of ~10 days that it took to resolve the issue some customers experienced issues seeing performance data in Azure monitor when Azure mode is selected and 5xx errors while viewing performance data on browser.
- Root Cause: The failure was due to one of the back end services turning unhealthy after a deployment.
- Incident Timeline: 2/15 through 2/25, 18:40 UTC
-Anupama
Update: Thursday, 25 February 2021 04:22 UTC
Root cause has been isolated to service turning unhealthy after incorrect deployment which was impacting performance data view in Azure monitor when Azure mode is selected. To address this issue engineers are rolling back the service to previous healthy state. Some customers who have VM's sending data to workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected and 5xx errors while viewing performance data on browser.
Root cause has been isolated to service turning unhealthy after incorrect deployment which was impacting performance data view in Azure monitor when Azure mode is selected. To address this issue engineers are rolling back the service to previous healthy state. Some customers who have VM's sending data to workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected and 5xx errors while viewing performance data on browser.
- Work Around: None
- Next Update: Before 02/26 04:30 UTC
Update: Thursday, 25 February 2021 01:28 UTC
Root cause has been isolated to service turning unhealthy after incorrect deployment which was impacting performance data view in Azure monitor when Azure mode is selected. To address this issue engineers are rolling back the service to previous healthy state. Some customers who have VM's sending data to workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected and 5xx errors while viewing performance data on browser.
Root cause has been isolated to service turning unhealthy after incorrect deployment which was impacting performance data view in Azure monitor when Azure mode is selected. To address this issue engineers are rolling back the service to previous healthy state. Some customers who have VM's sending data to workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected and 5xx errors while viewing performance data on browser.
- Work Around: None
- Next Update: Before 02/25 05:30 UTC
Update: Thursday, 25 February 2021 00:01 UTC
Root cause has been isolated to service turning unhealthy after incorrect deployment which was impacting performance data view in Azure monitor when Azure mode is selected. To address this issue we are rolling out a fix to the service. Some customers who have VM's sending data to workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected and 5xx errors while viewing performance data on browser.
Root cause has been isolated to service turning unhealthy after incorrect deployment which was impacting performance data view in Azure monitor when Azure mode is selected. To address this issue we are rolling out a fix to the service. Some customers who have VM's sending data to workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected and 5xx errors while viewing performance data on browser.
- Work Around: None
- Next Update: Before 02/25 02:30 UTC
Update: Wednesday, 24 February 2021 22:01 UTC
We continue to investigate issues within Azure monitor for VM's. Preliminary investigation showed that the root cause was due to one of the back end component becoming inaccessible after a scheduled drill/test in EastUS2EUAP region. Further investigation shows that the root cause is due to regression that happened after a deployment on one of the back end services. Some customers who have VM's sending data to workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected. Customers might also see 5xx errors while viewing the performance data on the portal.
- Work Around: None
- Next Update: Before 02/25 00:30 UTC
Update: Wednesday, 24 February 2021 12:02 UTC
We continue to investigate issues within Service Map. Root cause is due to one of the backend component becoming inaccessible after a scheduled drill/test in EastUS2EUAP region. Some customers who have VM's sending data to EastUS2EUAP workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected.
We continue to investigate issues within Service Map. Root cause is due to one of the backend component becoming inaccessible after a scheduled drill/test in EastUS2EUAP region. Some customers who have VM's sending data to EastUS2EUAP workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected.
- Work Around: None
- Next Update: Before 02/25 00:30 UTC
Update: Wednesday, 24 February 2021 01:32 UTC
We continue to investigate issues within Service Map. Root cause is due to one of the backend component becoming inaccessible after a scheduled drill/test in EastUS2EUAP region. Some customers who have VM's sending data to EastUS2EUAP workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected.
We continue to investigate issues within Service Map. Root cause is due to one of the backend component becoming inaccessible after a scheduled drill/test in EastUS2EUAP region. Some customers who have VM's sending data to EastUS2EUAP workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected.
- Work Around: None
- Next Update: Before 02/24 14:00 UTC
Update: Tuesday, 23 February 2021 21:22 UTC
We continue to investigate issues within Service map. Root cause is due to one of the backend component becoming inaccessible after a scheduled drill/test in EastUS2EUAP region. Some customers who have VM's sending data to EastUS2EUAP workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected. The issue started at 16:00 UTC
We continue to investigate issues within Service map. Root cause is due to one of the backend component becoming inaccessible after a scheduled drill/test in EastUS2EUAP region. Some customers who have VM's sending data to EastUS2EUAP workspaces, continue to experience issues seeing performance data in Azure monitor when Azure mode is selected. The issue started at 16:00 UTC
- Work Around: None
- Next Update: Before 02/23 23:30 UTC
Updated Feb 25, 2021
Version 8.0