Data loss - Issues uploading profiler traces and snapshots to App Insights - 01/10 - Mitigated

Published Feb 19 2019 08:41 AM 322 Views
First published on MSDN on Jan 10, 2019
Final Update: Thursday, 10 January 2019 21:30 UTC

We've confirmed that all systems are back to normal with no customer impact as of 1/10, 9:45 PM UTC. Our logs show the incident started on 1/10, 1:00 PM UTC and that during the 8 hours that it took to resolve the issue few of customers might have experienced failures with upload in profiler traces to App Insights.
  • Root Cause: The failure was due to spike in storage account usage which resulted profiler traces upload failures.
  • Incident Timeline: 8 Hours-  1/10, 1:00 PM UTC through 1/10, 9:45 PM UTC
We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-v-anurud
Update: Thursday, 10 January 2019 19:48 UTC

Root cause has been isolated to spike in storage account which was impacting uploading profiler traces. To address this issue we failed over the problematic storage account. Profiler service is now being monitored.
  • Work Around: none
  • Next Update: Before 01/10 22:00 UTC
-v-anurud
Initial Update: Thursday, 10 January 2019 18:52 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may not be able to upload profiler traces and snapshots to App insights.
  • Work Around: none
  • Next Update: Before 01/10 21:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-v-anurud
Version history
Last update:
‎Feb 19 2019 08:41 AM
Updated by: