Final Update: Saturday, 22 August 2020 10:24 UTC
We've confirmed that all systems are back to normal with no customer impact as of 8/22, 10:11 UTC. Our logs show the incident started on 8/20, 18:00 UTC and that during the 2 days, 16 hours and 11minutes that it took to resolve the issue, a very small percentage of customers in all public regions may have experience data gaps with the below data types:
We've confirmed that all systems are back to normal with no customer impact as of 8/22, 10:11 UTC. Our logs show the incident started on 8/20, 18:00 UTC and that during the 2 days, 16 hours and 11minutes that it took to resolve the issue, a very small percentage of customers in all public regions may have experience data gaps with the below data types:
NETWORK_MONITORING_BLOB
ONPREM_IIS_BLOB
CUSTOM_LOG
WAD_EVENT_TABLE
ETW_EVENT_TABLE
PAASV2_TABLE
PAASV2_RELIABLE_SERVICE_TABLE
PAASV2_RELIABLE_ACTOR_TABLE
WAD_IIS_BLOB
LINUX_SYS_LOGS_TABLE
WAD_SECURITY_EVENT_TABLE
WDAV_THREAT_TABLE
WAAS_DEPLOYMENT_STATUS_TABLE
WUDO_AGGREGATED_STATUS_TABLE
WAAS_UPDATE_STATUS_TABLE
WDAV_STATUS_TABLE
WAAS_INSIDER_STATUS_TABLE
WUDO_STATUS_TABLE
MAOfficeMacroHealthIssues_TABLE
MA_OFFICEADDIN_TABLE
MAWindowsCurrencyAssessmentDailyCounts_TABLE
MA_APPLICATIONINSTANCE_TABLE
MA_PROPOSEDPILOTDEVICES_TABLE
MAOfficeCurrencyAssessmentDailyCounts_TABLE
MAWindowsBuildInfo_TABLE
MAApplicationHealthAlternativeVersions_TABLE
MA_OFFICEADDINREADINESS_TABLE
MA_APPLICATION_TABLE
MADevicePnPHealthIssues_TABLE
MAWindowsSysReqInstanceReadiness_TABLE
MAApplicationHealth_TABLE
MADeviceNotEnrolled_TABLE
MAOfficeMacroIssueReadiness_TABLE
MA_DRIVERREADINESS_TABLE
MAOfficeAddinHealthIssues_TABLE
MAWindowsCurrencyAssessment_TABLE
MAOfficeSuiteInstance_TABLE
MAOfficeMacroIssueInstanceReadiness_TABLE
MAApplicationHealthIssues_TABLE
MA_OFFICEADDININSTANCE_TABLE
MA_OFFICEAPP_TABLE
MAOfficeBuildInfo_TABLE
MAOfficeSuite_TABLE
MAOfficeAddinInstanceReadiness_TABLE
MA_DEVICE_TABLE
MADriverInstanceReadiness_TABLE
MAOfficeMacroHealth_TABLE
MA_APPLICATIONREADINESS_TABLE
MAOfficeDeploymentStatus_TABLE
MA_DEVICEREADINESS_TABLE
MA_OFFICEAPPINSTANCE_TABLE
MAOfficeAddinHealth_TABLE
MAOfficeAppInstanceHealth_TABLE
MADevicePnPHealthAlternativeVersions_TABLE
MAOfficeAppHealth_TABLE
MA_DEPLOYMENTPLAN_TABLE
MAOfficeMacroSummary_TABLE
MAApplicationInstanceReadiness_TABLE
MAOfficeCurrencyAssessment_TABLE
MADevicePnPHealth_TABLE
MAWindowsDeploymentStatus_TABLE
MA_OFFICEAPPREADINESS_TABLE
OFFICEACTIVITY_RESTAPI
MAOfficeAddinEntityHealthInfo_TABLE
MAOfficeMacroGlobalHealthInfo_TABLE
SHOEBOX_NETWORK_EVENT_LOG_BLOB
SHOEBOX_NETWORK_COUNTER_LOG_BLOB
ONPREM_IIS_BLOB
CUSTOM_LOG
WAD_EVENT_TABLE
ETW_EVENT_TABLE
PAASV2_TABLE
PAASV2_RELIABLE_SERVICE_TABLE
PAASV2_RELIABLE_ACTOR_TABLE
WAD_IIS_BLOB
LINUX_SYS_LOGS_TABLE
WAD_SECURITY_EVENT_TABLE
WDAV_THREAT_TABLE
WAAS_DEPLOYMENT_STATUS_TABLE
WUDO_AGGREGATED_STATUS_TABLE
WAAS_UPDATE_STATUS_TABLE
WDAV_STATUS_TABLE
WAAS_INSIDER_STATUS_TABLE
WUDO_STATUS_TABLE
MAOfficeMacroHealthIssues_TABLE
MA_OFFICEADDIN_TABLE
MAWindowsCurrencyAssessmentDailyCounts_TABLE
MA_APPLICATIONINSTANCE_TABLE
MA_PROPOSEDPILOTDEVICES_TABLE
MAOfficeCurrencyAssessmentDailyCounts_TABLE
MAWindowsBuildInfo_TABLE
MAApplicationHealthAlternativeVersions_TABLE
MA_OFFICEADDINREADINESS_TABLE
MA_APPLICATION_TABLE
MADevicePnPHealthIssues_TABLE
MAWindowsSysReqInstanceReadiness_TABLE
MAApplicationHealth_TABLE
MADeviceNotEnrolled_TABLE
MAOfficeMacroIssueReadiness_TABLE
MA_DRIVERREADINESS_TABLE
MAOfficeAddinHealthIssues_TABLE
MAWindowsCurrencyAssessment_TABLE
MAOfficeSuiteInstance_TABLE
MAOfficeMacroIssueInstanceReadiness_TABLE
MAApplicationHealthIssues_TABLE
MA_OFFICEADDININSTANCE_TABLE
MA_OFFICEAPP_TABLE
MAOfficeBuildInfo_TABLE
MAOfficeSuite_TABLE
MAOfficeAddinInstanceReadiness_TABLE
MA_DEVICE_TABLE
MADriverInstanceReadiness_TABLE
MAOfficeMacroHealth_TABLE
MA_APPLICATIONREADINESS_TABLE
MAOfficeDeploymentStatus_TABLE
MA_DEVICEREADINESS_TABLE
MA_OFFICEAPPINSTANCE_TABLE
MAOfficeAddinHealth_TABLE
MAOfficeAppInstanceHealth_TABLE
MADevicePnPHealthAlternativeVersions_TABLE
MAOfficeAppHealth_TABLE
MA_DEPLOYMENTPLAN_TABLE
MAOfficeMacroSummary_TABLE
MAApplicationInstanceReadiness_TABLE
MAOfficeCurrencyAssessment_TABLE
MADevicePnPHealth_TABLE
MAWindowsDeploymentStatus_TABLE
MA_OFFICEAPPREADINESS_TABLE
OFFICEACTIVITY_RESTAPI
MAOfficeAddinEntityHealthInfo_TABLE
MAOfficeMacroGlobalHealthInfo_TABLE
SHOEBOX_NETWORK_EVENT_LOG_BLOB
SHOEBOX_NETWORK_COUNTER_LOG_BLOB
- Root Cause: The failure was due to a bad configuration in a recent build.
- Incident Timeline: 2 days, 16 Hours & 11 minutes - 8/20, 18:00 UTC through 8/22, 10:11 UTC
-Eric Singleton
Update: Saturday, 22 August 2020 08:33 UTC
Engineers are in the process of finalizing the rollback of the build which was impacting the ingestion time of certain data types in all public regions.
Engineers are in the process of finalizing the rollback of the build which was impacting the ingestion time of certain data types in all public regions.
- Work Around: None
- Next Update: Before 08/22 11:00 UTC
Update: Saturday, 22 August 2020 06:22 UTC
Engineers are still in the process of rolling back the build which was impacting the ingestion time of certain data types to a known good build.
Engineers are still in the process of rolling back the build which was impacting the ingestion time of certain data types to a known good build.
- Work Around: None
- Next Update: Before 08/22 08:30 UTC
Update: Saturday, 22 August 2020 03:45 UTC
Root cause has been isolated to a bad configuration in a recent build which was impacting the ingestion time of certain data types causing possible data gaps and unexpectedly latent log ingestion for some customers . To address this issue we are rolling back to a known good build.
Root cause has been isolated to a bad configuration in a recent build which was impacting the ingestion time of certain data types causing possible data gaps and unexpectedly latent log ingestion for some customers . To address this issue we are rolling back to a known good build.
- Work Around: None
- Next Update: Before 08/22 06:00 UTC
Initial Update: Saturday, 22 August 2020 03:03 UTC
We are aware of issues within Log Analytics and are actively investigating. Some customers in all public regions may experience data gaps for certain data types.
-Jeff
We are aware of issues within Log Analytics and are actively investigating. Some customers in all public regions may experience data gaps for certain data types.
- Work Around: None
- Next Update: Before 08/22 05:30 UTC
-Jeff
Updated Aug 22, 2020
Version 5.0