Home
%3CLINGO-SUB%20id%3D%22lingo-sub-828736%22%20slang%3D%22en-US%22%3EExperiencing%20classic%20metric%20alerting%20failure%20in%20Application%20Insights%20-%2008%2F29%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-828736%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CU%3EFinal%20Update%3C%2FU%3E%3A%20Thursday%2C%2029%20August%202019%2009%3A21%20UTC%3CBR%20%2F%3E%3CBR%20%2F%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%208%2F28%2C%2000%3A30%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%208%2F26%2C%2020%3A00%20UTC%20and%20that%20during%20the%20~%2028%20hours%20and%2030%20minutes%20that%20it%20took%20to%20resolve%20the%20issue%20some%20customers%20would%20not%20have%20received%20an%20email%20notification%20for%20classic%20AI%20metric%20alerts.%20This%20would%20have%20impacted%20some%20platform%20metrics%20and%20all%20the%20custom%20metrics.%20All%20other%20functionalities%20were%20working%20as%20expected.%26nbsp%3B%3CBR%20%2F%3E%3CUL%3E%3CLI%3E%3CU%3ERoot%20Cause%3C%2FU%3E%3A%20The%20failure%20was%20due%20to%20a%20recent%20change%20in%20one%20of%20our%20dependent%20backend%20service%20responsible%20for%20handling%20email%20notifications.%3C%2FLI%3E%3CLI%3E%3CU%3EIncident%20Timeline%3C%2FU%3E%3A%2028%20Hours%20%26amp%3B%2030%20minutes%20-%208%2F26%2C%2020%3A00%20UTC%26nbsp%3Bthrough%208%2F28%2C%2000%3A30%20UTC%3CBR%20%2F%3E%3C%2FLI%3E%3C%2FUL%3EWe%20understand%20that%20customers%20rely%20on%20Application%20Insights%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3CBR%20%2F%3E%3CBR%20%2F%3E-Varun%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-828736%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EApplication%20Insights%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Thursday, 29 August 2019 09:21 UTC

We've confirmed that all systems are back to normal with no customer impact as of 8/28, 00:30 UTC. Our logs show the incident started on 8/26, 20:00 UTC and that during the ~ 28 hours and 30 minutes that it took to resolve the issue some customers would not have received an email notification for classic AI metric alerts. This would have impacted some platform metrics and all the custom metrics. All other functionalities were working as expected. 
  • Root Cause: The failure was due to a recent change in one of our dependent backend service responsible for handling email notifications.
  • Incident Timeline: 28 Hours & 30 minutes - 8/26, 20:00 UTC through 8/28, 00:30 UTC
We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Varun