%3CLINGO-SUB%20id%3D%22lingo-sub-1649173%22%20slang%3D%22en-US%22%3EExperiencing%20Alerting%20failure%20for%20Azure%20Sentinel%20-%2009%2F09%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1649173%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Wednesday%2C%2009%20September%202020%2017%3A18%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2009%2F09%2C16%3A53%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2009%2F06%2C%2007%3A00%20UTC%20and%20that%20during%20the%203%20days%2C%209%20hours%20and%2053%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%20small%20set%20of%20customers%20using%20Azure%20Sentinel%20and%20Log%20Search%20Alert%20may%20have%20experienced%20failures%20in%20running%20alert%20rules%20which%20caused%20alerts%20to%20not%20be%20published%20to%20the%20workspace.%20Azure%20Sentinel%20retries%20failed%20queries%2C%20so%20most%20of%20the%20queries%20should%20eventually%20succeed.%26nbsp%3B%3CBR%20%2F%3E%3CUL%3E%0A%20%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20dependency%20on%20one%20of%20the%20backend%20services.%26nbsp%3B%3C%2FLI%3E%0A%20%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%203%20Days%2C%209%20Hours%20%26amp%3B%2053%20minutes%20-%20%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20rgb(255%2C%20255%2C%20255)%3B%20color%3A%20rgb(0%2C%200%2C%200)%3B%20font-family%3A%20%26quot%3BHelvetica%20Neue%26quot%3B%2CHelvetica%2CArial%2Csans-serif%3B%20font-size%3A%2014px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20400%3B%20letter-spacing%3A%20normal%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3E09%2F06%2C%2007%3A00%3C%2FSPAN%3E%20UTC%20through%20%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20rgb(255%2C%20255%2C%20255)%3B%20color%3A%20rgb(0%2C%200%2C%200)%3B%20font-family%3A%20%26quot%3BHelvetica%20Neue%26quot%3B%2CHelvetica%2CArial%2Csans-serif%3B%20font-size%3A%2014px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20400%3B%20letter-spacing%3A%20normal%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3E09%2F09%2C%2016%3A53%3C%2FSPAN%3E%20UTC%3C%2FLI%3E%0A%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Alert%20rules%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Jayadev%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EInitial%20Update%3C%2FU%3E%3A%20Wednesday%2C%2009%20September%202020%2015%3A55%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20are%20aware%20of%20issues%20within%20Azure%20Sentinel%20Service%20and%20are%20actively%20investigating.%20Some%20customers%20may%20see%20the%20alert%20rules%20failing%20and%20will%20hence%20may%20not%20able%20to%20publish%20the%20alert%20to%20the%20workspace.%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2009%2F09%2020%3A00%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Mohini%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1649173%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3ELog%20Search%20Alerts%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ESmart%20Diagnostics%20Alerts%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Wednesday, 09 September 2020 17:18 UTC

We've confirmed that all systems are back to normal with no customer impact as of 09/09,16:53 UTC. Our logs show the incident started on 09/06, 07:00 UTC and that during the 3 days, 9 hours and 53 minutes that it took to resolve the issue small set of customers using Azure Sentinel and Log Search Alert may have experienced failures in running alert rules which caused alerts to not be published to the workspace. Azure Sentinel retries failed queries, so most of the queries should eventually succeed. 
  • Root Cause: The failure was due to dependency on one of the backend services. 
  • Incident Timeline: 3 Days, 9 Hours & 53 minutes - 09/06, 07:00 UTC through 09/09, 16:53 UTC
We understand that customers rely on Alert rules as a critical service and apologize for any impact this incident caused.

-Jayadev

Initial Update: Wednesday, 09 September 2020 15:55 UTC

We are aware of issues within Azure Sentinel Service and are actively investigating. Some customers may see the alert rules failing and will hence may not able to publish the alert to the workspace.
  • Work Around: None
  • Next Update: Before 09/09 20:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Mohini