Home
%3CLINGO-SUB%20id%3D%22lingo-sub-650760%22%20slang%3D%22en-US%22%3E%5BFairFax%5D%20-%20Experiencing%20Data%20Loss%20Issue%20in%20workspaces%20for%20Log%20Analytics%20-%2005%2F27%20-%20Mitigated%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-650760%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CFONT%20face%3D%22%26quot%3BHelvetica%20Neue%26quot%3B%2CHelvetica%2CArial%2Csans-serif%22%3EFinal%20Update%3A%20Monday%2C%2027%20May%202019%2023%3A01%20UTC%3C%2FFONT%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CFONT%20face%3D%22%26quot%3BHelvetica%20Neue%26quot%3B%2CHelvetica%2CArial%2Csans-serif%22%3E%3CBR%20%2F%3E%3C%2FFONT%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CFONT%20face%3D%22%26quot%3BHelvetica%20Neue%26quot%3B%2CHelvetica%2CArial%2Csans-serif%22%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2005%2F27%2C%2023%3A10%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2005%2F27%2C%2013%3A05%20UTC%20and%20that%20during%20the%209%20hours%2054%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%20all%20FairFax(USGov)%20customers%20may%20have%20observed%20missing%20records%20in%20their%20workspaces%20for%20specific%20set%20of%20data%20types.%3CBR%20%2F%3ERoot%20Cause%3A%20The%20failure%20was%20due%20to%20an%20issue%20in%20our%20one%20of%20our%20backend%20services.%3CBR%20%2F%3EIncident%20Timeline%3A%209%20Hours%20%26amp%3B%2054%20minutes%20-%2005%2F27%2C%2013%3A05%20UTC%20through%2005%2F27%2C%2022%3A59%20UTC%3CBR%20%2F%3EWe%20understand%20that%20customers%20rely%20on%20Azure%20Log%20Analytics%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3C%2FFONT%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CFONT%20face%3D%22%26quot%3BHelvetica%20Neue%26quot%3B%2CHelvetica%2CArial%2Csans-serif%22%3E%3CBR%20%2F%3E%3C%2FFONT%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CFONT%20face%3D%22%26quot%3BHelvetica%20Neue%26quot%3B%2CHelvetica%2CArial%2Csans-serif%22%3E-Sindhu%3C%2FFONT%3E%3CU%3E%3C%2FU%3E%3CU%3E%3C%2FU%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3EUpdate%3A%20Monday%2C%2027%20May%202019%2020%3A48%20UTC%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3EWe%20continue%20to%20investigate%20issues%20within%20Log%20Analytics.%20Root%20cause%20is%20not%20fully%20understood%20at%20this%20time.%20Some%20customers%20continue%20to%20observe%20missing%20records%20in%20thier%20workspaces%20for%20a%20specific%20set%20of%20data%20types.%20Our%20initial%20findings%20indicate%20that%20the%20problem%20began%20at%2005%2F27%2013%3A05%20UTC.%20We%20currently%20have%20no%20estimate%20for%20resolution.%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3EWork%20Around%3A%20None%3CBR%20%2F%3ENext%20Update%3A%20Before%2005%2F27%2023%3A00%20UTC%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3EImpacted%20Data%20Types%3A%26nbsp%3B%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3EHEALTH_ASSESSMENT_BLOB%3CBR%20%2F%3EDNS_ANALYTICS_BLOB%3CBR%20%2F%3EAZURE_METRICS_EMBEDDEDDATA%3CBR%20%2F%3ELINUX_SYSLOGS_BLOB%3CBR%20%2F%3EDNS_DYNAMIC_BLOB%3CBR%20%2F%3EPROTECTION_STATUS_BLOB%3CBR%20%2F%3EOPERATION_BLOB%3CBR%20%2F%3ECONTAINER_LOG_BLOB%3CBR%20%2F%3ECONTAINER_INVENTORY_BLOB%3CBR%20%2F%3ECONTAINER_IMAGE_INVENTORY_BLOB%3CBR%20%2F%3ECONTAINER_SERVICE_LOG_BLOB%3CBR%20%2F%3EUSAGE_METERING%3CBR%20%2F%3EDNS_AUDIT_BLOB%3CBR%20%2F%3ECOMPUTER_GROUP_BLOB%3CBR%20%2F%3EOMSALERTS_BLOB%3CBR%20%2F%3EOFFICEACTIVITY_RESTAPI%3CBR%20%2F%3EKUBE_SERVICES_BLOB%3CBR%20%2F%3EKUBE_POD_INVENTORY_BLOB%3CBR%20%2F%3EKUBE_NODE_INVENTORY_BLOB%3CBR%20%2F%3EDNS_INVENTORY_BLOB%3CBR%20%2F%3EALERT_SUBSCRIPTION_BLOB%3CBR%20%2F%3EKUBE_EVENTS_BLOB%3CBR%20%2F%3EUPDATES_RUN_PROGRESS_BLOB%3CBR%20%2F%3ESQL_ASSESSMENT_RECOMMENDATION_BLOB%3CBR%20%2F%3EAD_REPLICATION_BLOB%3CBR%20%2F%3EAD_ASSESSMENT_RECOMMENDATION_BLOB%3CBR%20%2F%3EONPREM_IIS_BLOB%3CBR%20%2F%3EPAASV2_TABLE%3CBR%20%2F%3EPAASV2_RELIABLE_SERVICE_TABLE%3CBR%20%2F%3EPAASV2_RELIABLE_ACTOR_TABLE%3CBR%20%2F%3EETW_EVENT_TABLE%3CBR%20%2F%3EAUDIT_LOG_REST_API%3CBR%20%2F%3ECOMPAT_SYSREQISSUE_TABLE%3CBR%20%2F%3EAZURE_METRICS_EMBEDDEDDATA%3CBR%20%2F%3EWAD_IIS_BLOB%3CBR%20%2F%3ECOMPAT_APP_TABLE%3CBR%20%2F%3ECOMPAT_DRIVER_TABLE%3CBR%20%2F%3ECOMPAT_IESITEDISCOVERY_TABLE%3CBR%20%2F%3ELINUX_SYS_LOGS_TABLE%3CBR%20%2F%3ECOMPAT_UPGRADEDCOMPUTER_TABLE%3CBR%20%2F%3ECOMPAT_COMPUTER_TABLE%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CBR%20%2F%3E-Sindhu%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CFONT%20face%3D%22%26quot%3BHelvetica%20Neue%26quot%3B%2CHelvetica%2CArial%2Csans-serif%22%3EInitial%20Update%3A%20Monday%2C%2027%20May%202019%2018%3A50%20UTC%3C%2FFONT%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CFONT%20face%3D%22%26quot%3BHelvetica%20Neue%26quot%3B%2CHelvetica%2CArial%2Csans-serif%22%3EWe%20are%20aware%20of%20issues%20within%20Log%20Analytics%20and%20are%20actively%20investigating.%26nbsp%3B%20All%20customers%20in%20Fairfax(USGov)%20region%20may%20observe%20missing%20records%20in%20their%20workspaces%20for%20specific%20set%20of%20data%20types.%3C%2FFONT%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CFONT%20face%3D%22%26quot%3BHelvetica%20Neue%26quot%3B%2CHelvetica%2CArial%2Csans-serif%22%3EWork%20Around%3A%20None%3CBR%20%2F%3ENext%20Update%3A%20Before%2005%2F27%2021%3A00%20UTC%20%3CBR%20%2F%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Sindhu%3C%2FFONT%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-650760%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Log%20Analytics%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Monday, 27 May 2019 23:01 UTC

We've confirmed that all systems are back to normal with no customer impact as of 05/27, 23:10 UTC. Our logs show the incident started on 05/27, 13:05 UTC and that during the 9 hours 54 minutes that it took to resolve the issue all FairFax(USGov) customers may have observed missing records in their workspaces for specific set of data types.
Root Cause: The failure was due to an issue in our one of our backend services.
Incident Timeline: 9 Hours & 54 minutes - 05/27, 13:05 UTC through 05/27, 22:59 UTC
We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused.

-Sindhu

Update: Monday, 27 May 2019 20:48 UTC

We continue to investigate issues within Log Analytics. Root cause is not fully understood at this time. Some customers continue to observe missing records in thier workspaces for a specific set of data types. Our initial findings indicate that the problem began at 05/27 13:05 UTC. We currently have no estimate for resolution.
Work Around: None
Next Update: Before 05/27 23:00 UTC

Impacted Data Types: 
HEALTH_ASSESSMENT_BLOB
DNS_ANALYTICS_BLOB
AZURE_METRICS_EMBEDDEDDATA
LINUX_SYSLOGS_BLOB
DNS_DYNAMIC_BLOB
PROTECTION_STATUS_BLOB
OPERATION_BLOB
CONTAINER_LOG_BLOB
CONTAINER_INVENTORY_BLOB
CONTAINER_IMAGE_INVENTORY_BLOB
CONTAINER_SERVICE_LOG_BLOB
USAGE_METERING
DNS_AUDIT_BLOB
COMPUTER_GROUP_BLOB
OMSALERTS_BLOB
OFFICEACTIVITY_RESTAPI
KUBE_SERVICES_BLOB
KUBE_POD_INVENTORY_BLOB
KUBE_NODE_INVENTORY_BLOB
DNS_INVENTORY_BLOB
ALERT_SUBSCRIPTION_BLOB
KUBE_EVENTS_BLOB
UPDATES_RUN_PROGRESS_BLOB
SQL_ASSESSMENT_RECOMMENDATION_BLOB
AD_REPLICATION_BLOB
AD_ASSESSMENT_RECOMMENDATION_BLOB
ONPREM_IIS_BLOB
PAASV2_TABLE
PAASV2_RELIABLE_SERVICE_TABLE
PAASV2_RELIABLE_ACTOR_TABLE
ETW_EVENT_TABLE
AUDIT_LOG_REST_API
COMPAT_SYSREQISSUE_TABLE
AZURE_METRICS_EMBEDDEDDATA
WAD_IIS_BLOB
COMPAT_APP_TABLE
COMPAT_DRIVER_TABLE
COMPAT_IESITEDISCOVERY_TABLE
LINUX_SYS_LOGS_TABLE
COMPAT_UPGRADEDCOMPUTER_TABLE
COMPAT_COMPUTER_TABLE

-Sindhu

Initial Update: Monday, 27 May 2019 18:50 UTC
We are aware of issues within Log Analytics and are actively investigating.  All customers in Fairfax(USGov) region may observe missing records in their workspaces for specific set of data types.
Work Around: None
Next Update: Before 05/27 21:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Sindhu