Blog Post

Azure Monitor Status Archive
2 MIN READ

Experiencing Data Latency issue in Azure Portal for Many Data Types - 09/03 - Resolved

Azure-Monitor-Team's avatar
Azure-Monitor-Team
Former Employee
Sep 03, 2020
Final Update: Thursday, 03 September 2020 22:36 UTC

We've confirmed that all systems are back to normal with no customer impact as of 09/03, 21:56 UTC. Our logs show the incident started on 09/03, 09:30 UTC and that during the 12 hours and 26 minutes that it took to resolve the issue some customers might have experienced intermittent data latency, data gaps and incorrect alert activation in West Europe region.
  • Root Cause: The issue was due to hardware failure in one of the region's data centers affecting the network.
  • Incident Timeline: 12 Hours & 26 minutes - 09/03, 09:30 UTC through 09/03, 21:56 UTC
We understand that customers rely on Application Insights and Azure Log Analytics as critical services and apologize for any impact this incident caused.

-Saika
Update: Thursday, 03 September 2020 20:00 UTC

Root cause has been isolated to a hardware failure in one of the region's data centers affecting the network and systems are still catching up to get back to normal. Some customers may still continue to experience intermittent data latency, data gaps and incorrect alert activation in West Europe region.
  • Next Update: Before 09/04 00:00 UTC
-Saika
Update: Thursday, 03 September 2020 15:30 UTC

We continue to investigate issues within Application Insights and Azure Log Analytics. Some customers continue to experience intermittent data latency, data gaps and incorrect alert activation in West Europe region. The initial findings suggest that the issue began at 09/03 ~09:30 UTC.
  • Work Around: None
  • Next Update: Before 09/03 20:30 UTC
-Sandeep
Update: Thursday, 03 September 2020 12:38 UTC

We continue to investigate issues within Application Insights and Azure Log Analytics in West Europe region. Root cause is not fully understood at this time. Some customers continue to experience intermittent data latency, data gaps and incorrect alert activation.  We are working to establish the start time for the issue, initial findings indicate that the problem began at 09/03 ~09:30 UTC. We currently have no estimate for resolution.
  • Work Around: None
  • Next Update: Before 09/03 15:00 UTC
-Rama
Initial Update: Thursday, 03 September 2020 10:34 UTC

We are aware of issues with Data Latency within Application Insights and Log Analytics in West Europe region and are actively investigating. Some customers may experience intermittent data latency, data gaps and incorrect alert activation.
  • Work Around: None
  • Next Update: Before 09/03 14:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Sandeep

Updated Sep 03, 2020
Version 9.0
application insights
Azure Log Analytics

4 Comments

  • Nik Smit's avatar
    Nik Smit
    Copper Contributor
    Sep 04, 2020

    1) Just to echo the lack of transparency. Ironic considering this recent post:  https://azure.microsoft.com/en-gb/blog/advancing-the-outage-experience-automation-communication-and-transparency/   

     

    2) After a 12 hour outage, "The issue was due to hardware failure in one of the region's data centers affecting the network." is not really an ideal explanation.  We all assume you have redundant network hardware, so this is frankly not sufficient.   At least commit to a follow up blog post that explains it in more detail and the future mitigations?

  • loadedlouie27's avatar
    loadedlouie27
    Brass Contributor
    Sep 03, 2020

    I have 2 clients subscriptions that are having the same issue.

    No Logs since 09H20 UTC approx. 
    Do you guys have any clues on what's going on? 

     

    I do have logs but not from VM's, and app insights was also facing a delay. to the extend of availability tests also be impacted. 

     

    Any suggestions. ?

  • Aleksandras's avatar
    Aleksandras
    Copper Contributor
    Sep 03, 2020

    It's understandable there are issues some times. However if it's already an acknowledged issue and you are investigating it, then why there are still no updates neither on status.azure.com,  nor in "Service Health" in Azure Portal?

    It's often a hassle to find out if there are some Azure and/or infrastructure issues, or something with our own solutions..

    Not that it's usually acknowledged with a delay (when customers facing it usually already know about it, or reported it themselves), but it's also might be announced in a few different places, so that you need to monitor status.azure.com, Azure Portal, various forums, or even Tweeter or our own corporate resources.