Experiencing Availability test failure issue - 10/20 - Resolved

[Azure-Monitor-Team]

Final Update: Tuesday, 20 October 2020 17:17 UTC

We've confirmed that all systems are back to normal as of 10/20, 16:00 UTC. Our logs show the incident started on 10/15, 15:00 UTC and that during the time it took to resolve the issue  customers experienced latency up to 14 hours when performing service management operations - such as create, update, delete – for Availability tests.

• Root Cause: The failure was due to recent configuration changes that caused some instances of a backend service to reach operational threshold.
• Incident Timeline: 10/15, 15:00 UTC through 10/20, 16:00 UTC
  

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Chandar

---

Update: Tuesday, 20 October 2020 13:43 UTC

Root cause has been isolated and identified as a configuration change in one of the dependent service is causing the issue. Engineers are working on applying the mitigation. New or updated Availability test continue to be delayed. 

• Work Around: None
• Next Update: Before 10/20 18:00 UTC

-Sandeep

---

Update: Tuesday, 20 October 2020 09:09 UTC

New or updated Availability test continue to be delayed. To address this, engineers are currently working to identify changes in dependent services that is causing the issue. We currently have no estimate for resolution.

• Work Around: None
• Next Update: Before 10/20 13:30 UTC

-Sandeep

---

Update: Tuesday, 20 October 2020 06:19 UTC

We continue to investigate issues within Application Insights. Root cause is not fully understood at this time. Engineers are working on applying the mitigation. New or updated Availability test continue to be delayed.  We currently have no estimate for resolution.

• Work Around: None
• Next Update: Before 10/20 09:30 UTC

-Sandeep

---

Update: Tuesday, 20 October 2020 03:27 UTC

We continue to investigate issues within Application Insights. Root cause is not fully understood at this time, preliminary investigation points to a configuration issue in one of the components.. New or updated Availability test continue to be delayed. 

• Work Around:
• Next Update: Before 10/20 06:30 UTC

-chandar

---

Update: Tuesday, 20 October 2020 01:40 UTC

We continue to investigate issues within Application Insights. Root cause is not fully understood at this time. New or updated Availability test continue to be delayed in taking effect by up to 14 hours. Initial findings indicate that the problem began at 10/15~15:00 UTC. We currently have no estimate for resolution.

• Work Around:
• Next Update: Before 10/20 03:00 UTC

-chandar

---

Initial Update: Tuesday, 20 October 2020 00:27 UTC

We are aware of issues within Application Insights and are actively investigating. Newly created or updated Availability Tests do not take effect in executing the tests.

• Work Around:
• Next Update: Before 10/20 01:30 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-chandar

---