Final Update: Tuesday, 03 August 2021 07:01 UTC
We've confirmed that all systems are back to normal with no customer impact as of 08/03, 03:00 UTC. Our logs show the incident started on 08/01, 15:30 UTC and that during the 1 Day 11 Hours & 30 minutes that it took to resolve the issue. Some customers may have experienced delay in data ingestion, duplicated data and delayed or missed alerts in West US2 region
-Vyom
We've confirmed that all systems are back to normal with no customer impact as of 08/03, 03:00 UTC. Our logs show the incident started on 08/01, 15:30 UTC and that during the 1 Day 11 Hours & 30 minutes that it took to resolve the issue. Some customers may have experienced delay in data ingestion, duplicated data and delayed or missed alerts in West US2 region
- Root Cause: The failure was due to an issue with one of the dependent services.
- Incident Timeline: 1 Day 11 Hours & 30 minutes - 08/01, 15:30 UTC through 08/03, 03:00 UTC
-Vyom
Update: Tuesday, 03 August 2021 05:08 UTC
Root cause has been isolated to configuration change which was impacting the data ingestion in the Log Analytics Workspace in West US 2 region. Some customers may continue to experience missing alerts and data and we estimate it may take few more hours before all missing alerts and data to be made available.
Root cause has been isolated to configuration change which was impacting the data ingestion in the Log Analytics Workspace in West US 2 region. Some customers may continue to experience missing alerts and data and we estimate it may take few more hours before all missing alerts and data to be made available.
- Work Around: None
- Next Update: Before 08/03 11:30 UTC
Update: Monday, 02 August 2021 22:23 UTC
***** REPOSTING COMMS AS NO UPDATE FROM ENGINEERING TEAM ****
Root cause has been isolated to configuration change which was impacting the data ingestion in the Log Analytics Workspace in West US 2 region. Some customers may continue to experience missing alerts and data and we estimate it may take more hours before all missing alerts and data to be made available.
***** REPOSTING COMMS AS NO UPDATE FROM ENGINEERING TEAM ****
Root cause has been isolated to configuration change which was impacting the data ingestion in the Log Analytics Workspace in West US 2 region. Some customers may continue to experience missing alerts and data and we estimate it may take more hours before all missing alerts and data to be made available.
- Work Around: None
- Next Update: Before 08/03 04:30 UTC
Update: Monday, 02 August 2021 17:26 UTC
***** REPOSTING COMMS AS NO UPDATE FROM ENGINEERING TEAM ****
Root cause has been isolated to configuration change which was impacting the data ingestion in the Log Analytics Workspace in West US 2 region. Some customers may continue to experience missing alerts and data and we estimate it may take more hours before all missing alerts and data to be made available.
***** REPOSTING COMMS AS NO UPDATE FROM ENGINEERING TEAM ****
Root cause has been isolated to configuration change which was impacting the data ingestion in the Log Analytics Workspace in West US 2 region. Some customers may continue to experience missing alerts and data and we estimate it may take more hours before all missing alerts and data to be made available.
- Work Around: None
- Next Update: Before 08/02 22:00 UTC
Update: Monday, 02 August 2021 10:57 UTC
Root cause has been isolated to configuration change which was impacting the data ingestion in the Log Analytics Workspace in West US 2 region. Some customers may continue to experience missing alerts and data and we estimate it may take more hours before all missing alerts and data to be made available.
Root cause has been isolated to configuration change which was impacting the data ingestion in the Log Analytics Workspace in West US 2 region. Some customers may continue to experience missing alerts and data and we estimate it may take more hours before all missing alerts and data to be made available.
- Work Around: None
- Next Update: Before 08/02 17:00 UTC
Update: Monday, 02 August 2021 07:35 UTC
Root cause has been isolated to configuration change which was impacting the data ingestion in the Log Analytics Workspace in West US 2 region. Some customers may experience missing alerts and data and we estimate it may take few more hours before all missing alerts and data to be made available.
Root cause has been isolated to configuration change which was impacting the data ingestion in the Log Analytics Workspace in West US 2 region. Some customers may experience missing alerts and data and we estimate it may take few more hours before all missing alerts and data to be made available.
- Work Around: None
- Next Update: Before 08/02 12:00 UTC
Update: Monday, 02 August 2021 04:12 UTC
***** REPOSTING COMMS AS NO UPDATE FROM ENGINEERING TEAM ****
Root cause has been isolated to miss configuration of some of WUS2 log analytics backend clusters which was impacting the data ingestion leading to missing alerts and data. To address this issue we applied missing configuration to some of WUS2 Log analytics Backend servers. The service is processing the backlog ingestion data which should take ~2 hours. Some customers may experience missing alerts and data and we estimate several hours before all missing alerts and data to be made available.
***** REPOSTING COMMS AS NO UPDATE FROM ENGINEERING TEAM ****
Root cause has been isolated to miss configuration of some of WUS2 log analytics backend clusters which was impacting the data ingestion leading to missing alerts and data. To address this issue we applied missing configuration to some of WUS2 Log analytics Backend servers. The service is processing the backlog ingestion data which should take ~2 hours. Some customers may experience missing alerts and data and we estimate several hours before all missing alerts and data to be made available.
- Work Around: none
- Next Update: Before 08/02 03:30 UTC
Update: Sunday, 01 August 2021 23:14 UTC
Root cause has been isolated to miss configuration of some of WUS2 log analytics backend clusters which was impacting the data ingestion leading to missing alerts and data. To address this issue we applied missing configuration to some of WUS2 Log analytics Backend servers. The service is processing the backlog ingestion data which should take ~2 hours. Some customers may experience missing alerts and data and we estimate several hours before all missing alerts and data to be made available.
Root cause has been isolated to miss configuration of some of WUS2 log analytics backend clusters which was impacting the data ingestion leading to missing alerts and data. To address this issue we applied missing configuration to some of WUS2 Log analytics Backend servers. The service is processing the backlog ingestion data which should take ~2 hours. Some customers may experience missing alerts and data and we estimate several hours before all missing alerts and data to be made available.
- Work Around: none
- Next Update: Before 08/02 03:30 UTC
Initial Update: Sunday, 01 August 2021 18:27 UTC
We are aware of issues within Log Analytics and are actively investigating. Some customers may experience delay in data ingestion into Log Analytics in West US2 .
-Subhash
We are aware of issues within Log Analytics and are actively investigating. Some customers may experience delay in data ingestion into Log Analytics in West US2 .
- Work Around:
- Next Update: Before 08/01 22:30 UTC
-Subhash
Updated Aug 03, 2021
Version 9.0