Home
%3CLINGO-SUB%20id%3D%22lingo-sub-645346%22%20slang%3D%22en-US%22%3E%5BFairFax%5D-Data%20Ingestion%20and%20Alert%20Notification%20Issues%20in%20Azure%20portal%20-%2005%2F24%20-%20Mitigated%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-645346%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Saturday%2C%2025%20May%202019%2002%3A49%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2005%2F25%2C%202%3A35%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2005%2F24%2C%2020%3A50%20UTC%20and%20that%20during%20the%205%20hours%20and%2045%20min%20that%20it%20took%20to%20resolve%20the%20issue%20approximately%2031%25%20of%20customers%20in%20FairFax(USGov)%20experienced%20delay%20in%20Data%20Ingestion%20and%20also%20some%20customers%20experienced%20issues%20with%20alert%20notifications%20configured%20based%20on%26nbsp%3B%3CSPAN%20helvetica%3D%22%22%20neue%3D%22%22%3Ewebhooks%26nbsp%3Bin%20FairFax(USGov)%3C%2FSPAN%3E%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20issue%20with%20storage%20service.%3C%2FLI%3E%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%205%20Hours%20%26amp%3B%2045%20minutes%20-%2005%2F24%2C%2020%3A50%26nbsp%3BUTC%20through%2005%2F25%2C%202%3A35%26nbsp%3BUTC%3CBR%20%2F%3E%3C%2FLI%3E%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Azure%20Log%20Analytics%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Venkat%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EUpdate%3C%2FU%3E%3A%20Saturday%2C%2025%20May%202019%2001%3A41%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20continue%20to%20investigate%20issues%20within%20Log%20Analytics%20and%20Alert%20Notification.Root%20cause%20has%20been%20isolated%20to%20a%20storage%20issue.%20Some%20Log%20Analytics%20customers%20continue%20to%20experience%20delay%20in%20Data%20Ingestion%20and%20also%20issues%20with%20alert%20notifications%20configured%20based%20on%26nbsp%3B%3CSPAN%20style%3D%22color%3A%20rgb(51%2C%2051%2C%2051)%3B%20font-family%3A%20SegoeUI%2C%20Lato%2C%20%22%20helvetica%3D%22%22%20neue%3D%22%22%3Ewebhooks%20in%20FairFax(USGov)%3C%2FSPAN%3E.%20We%20are%20working%20to%20establish%20the%20start%20time%20for%20the%20issue%2C%20initial%20findings%20indicate%20that%20the%20problem%20began%20at%2005%2F24%20~20%3A50%20UTC.%20We%20currently%20have%20no%20estimate%20for%20resolution.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20none%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2005%2F25%2006%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3E-Venkat%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Friday%2C%2024%20May%202019%2022%3A08%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Log%20Analytics%20and%20are%20actively%20investigating.%20Some%20customers%20may%20experience%20Data%20Ingestion%20delays%20in%20FairFax(USGov)%20region.%3C%2FDIV%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20none%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2005%2F25%2002%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Venkat%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-645346%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Log%20Analytics%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Saturday, 25 May 2019 02:49 UTC

We've confirmed that all systems are back to normal with no customer impact as of 05/25, 2:35 UTC. Our logs show the incident started on 05/24, 20:50 UTC and that during the 5 hours and 45 min that it took to resolve the issue approximately 31% of customers in FairFax(USGov) experienced delay in Data Ingestion and also some customers experienced issues with alert notifications configured based on webhooks in FairFax(USGov)
  • Root Cause: The failure was due to issue with storage service.
  • Incident Timeline: 5 Hours & 45 minutes - 05/24, 20:50 UTC through 05/25, 2:35 UTC
We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused.

-Venkat

Update: Saturday, 25 May 2019 01:41 UTC

We continue to investigate issues within Log Analytics and Alert Notification.Root cause has been isolated to a storage issue. Some Log Analytics customers continue to experience delay in Data Ingestion and also issues with alert notifications configured based on webhooks in FairFax(USGov). We are working to establish the start time for the issue, initial findings indicate that the problem began at 05/24 ~20:50 UTC. We currently have no estimate for resolution.
  • Work Around: none
  • Next Update: Before 05/25 06:00 UTC
-Venkat

Initial Update: Friday, 24 May 2019 22:08 UTC

We are aware of issues within Log Analytics and are actively investigating. Some customers may experience Data Ingestion delays in FairFax(USGov) region.

  • Work Around: none
  • Next Update: Before 05/25 02:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Venkat