%3CLINGO-SUB%20id%3D%22lingo-sub-1440281%22%20slang%3D%22en-US%22%3EExperiencing%20Data%20Latency%20in%20Azure%20portal%20for%20Activity%20Log%20Alerts%20-%2006%2F04%20-%20Investigating%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1440281%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22%22%3E%3CU%20style%3D%22font-size%3A%2014px%3B%22%3EInitial%20Update%3C%2FU%3E%3A%20Thursday%2C%2004%20June%202020%2013%3A18%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3E%3CSPAN%20style%3D%22color%3A%20rgb(51%2C%2051%2C%2051)%3B%20font-family%3A%20SegoeUI%2C%20Lato%2C%20%26quot%3BHelvetica%20Neue%26quot%3B%2C%20Helvetica%2C%20Arial%2C%20sans-serif%3B%22%3EWe%20are%20aware%20of%20issues%20within%20Activity%20Log%20Alerts%20and%20are%20actively%20investigating.%20Some%20customers%20may%20experience%20delayed%20alerts.%20Alerts%20would%20have%20eventually%20fired.%3C%2FSPAN%3E%3CBR%20style%3D%22color%3A%20rgb(51%2C%2051%2C%2051)%3B%20font-family%3A%20SegoeUI%2C%20Lato%2C%20%26quot%3BHelvetica%20Neue%26quot%3B%2C%20Helvetica%2C%20Arial%2C%20sans-serif%3B%22%20%2F%3E%3CUL%20style%3D%22font-size%3A%2014px%3B%20margin-bottom%3A%2012px%3B%20padding-left%3A%202.5em%3B%20list-style%3A%20outside%3B%20clear%3A%20left%3B%20color%3A%20rgb(51%2C%2051%2C%2051)%3B%20font-family%3A%20SegoeUI%2C%20Lato%2C%20%26quot%3BHelvetica%20Neue%26quot%3B%2C%20Helvetica%2C%20Arial%2C%20sans-serif%3B%22%3E%3C%2FUL%3E%3CUL%20style%3D%22font-size%3A%2014px%3B%22%3E%3CLI%3E%3CU%3EWork%20Around%3C%2FU%3E%3A%20None%3C%2FLI%3E%3CLI%3E%3CU%3ENext%20Update%3C%2FU%3E%3A%20Before%2006%2F04%2017%3A30%20UTC%3C%2FLI%3E%3C%2FUL%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3CBR%20%2F%3E-Anmol%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1440281%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EActivity%20Log%20Alerts%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1444361%22%20slang%3D%22en-US%22%3ERe%3A%20Experiencing%20Data%20Latency%20in%20Azure%20portal%20for%20Activity%20Log%20Alerts%20-%2006%2F04%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1444361%22%20slang%3D%22en-US%22%3E%3CP%3EHi%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20have%20an%20AIP%20environment%20and%20the%20activity%20logs%20sent%20by%20the%20Unified%20Labeling%20Clients%20currently%20arive%20with%20a%20delay%20of%20roundabout%20~12%20hours.%20Is%20there%20still%20an%20ongoing%20problem%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EBest%20regards%3C%2FP%3E%3C%2FLINGO-BODY%3E
Final Update: Thursday, 04 June 2020 19:20 UTC

We've confirmed that all systems are back to normal with no customer impact as of 6/4, 18:54 UTC. Our logs show the incident started on 6/4, 12:05 UTC and that during the 6 hours and 49 minutes that it took to resolve the issue customers experienced delayed log search alerts by up to 2.5 hours.
  • Root Cause: The failure was due to the service reaching an operating limit as a result of large unexpected spike in alert notifications across regions.
  • Incident Timeline: 6 Hours & 49 minutes - 6/4, 12:05 UTC through 6/4, 18:54 UTC
We understand that customers rely on Activity Log Alerts as a critical service and apologize for any impact this incident caused.

-Jeff

Update: Thursday, 04 June 2020 17:06 UTC

Root cause has been isolated to an extremely large spike in log search alerts across multiple regions which was impacting activity log alerts ability to process alerts in a timely manner. To address this issue we have scaled out the service and the primary incident causing the spike has been mitigated. Some customers may continue experience delayed log search alerts of up to 2 hours. We estimate 2-3 hours before timely log search alerts are being processed again.
  • Work Around: None
  • Next Update: Before 06/04 21:30 UTC
-Jeff

Initial Update: Thursday, 04 June 2020 13:18 UTC

We are aware of issues within Activity Log Alerts and are actively investigating. Some customers may experience delayed alerts. Alerts would have eventually fired.
    • Work Around: None
    • Next Update: Before 06/04 17:30 UTC
    We are working hard to resolve this issue and apologize for any inconvenience.
    -Anmol

    1 Comment
    Occasional Visitor

    Hi,

     

    I have an AIP environment and the activity logs sent by the Unified Labeling Clients currently arive with a delay of roundabout ~12 hours. Is there still an ongoing problem?

     

    Best regards