Data loss - Issues uploading profiler traces and snapshots to App Insights - 01/10 - Mitigated

[Azure-Monitor-Team]

First published on MSDN on Jan 10, 2019

Final Update: Thursday, 10 January 2019 21:30 UTC

We've confirmed that all systems are back to normal with no customer impact as of 1/10, 9:45 PM UTC. Our logs show the incident started on 1/10, 1:00 PM UTC and that during the 8 hours that it took to resolve the issue few of customers might have experienced failures with upload in profiler traces to App Insights.

• Root Cause: The failure was due to spike in storage account usage which resulted profiler traces upload failures.
• Incident Timeline: 8 Hours-  1/10, 1:00 PM UTC through 1/10, 9:45 PM UTC

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-v-anurud

Update: Thursday, 10 January 2019 19:48 UTC

Root cause has been isolated to spike in storage account which was impacting uploading profiler traces. To address this issue we failed over the problematic storage account. Profiler service is now being monitored.

• Work Around: none
• Next Update: Before 01/10 22:00 UTC

-v-anurud

Initial Update: Thursday, 10 January 2019 18:52 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may not be able to upload profiler traces and snapshots to App insights.

• Work Around: none
• Next Update: Before 01/10 21:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.
-v-anurud