Home
%3CLINGO-SUB%20id%3D%22lingo-sub-782330%22%20slang%3D%22en-US%22%3EAzure%20Monitor%20Metric%20Alert%20Failures%20in%20Azure%20Portal%20-%2007%2F31%20-%20Resolved%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-782330%22%20slang%3D%22en-US%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22%22%3E%3CDIV%20style%3D%22%22%3EFinal%20Update%3A%20Thursday%2C%2001%20August%202019%2000%3A27%20UTC%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3EWe've%20confirmed%20that%20all%20systems%20are%20back%20to%20normal%20with%20no%20customer%20impact%20as%20of%2001%2F08%2C%2000%3A27%20UTC.%20Our%20logs%20show%20the%20incident%20started%20on%2007%2F27%2C%2000%3A16%20UTC%20and%20that%20during%20the%20duration%20of%205%20days%20that%20it%20took%20to%20resolve%20the%20issue%20some%20customers%20globally%20may%20have%20experienced%20failures%20while%20performing%20CRUD%20operations(create%2C%20modify%20or%20delete)%20for%20alerting%20when%20network%20metric%20alerts%20are%20included%20in%20the%20configuration.%20Existing%20alerting%20will%20work%20as%20expected.%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3ERoot%20Cause%3A%20The%20failure%20was%20due%20to%20API%20contract%20issue%20between%20our%20dependent%20services%20which%20caused%20metric%20alert%20rule%20creation%20failure%20for%20Network%20Resource%20Type.%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3EIncident%20Timeline%3A%205%20days%26nbsp%3B%20-%2007%2F27%2C%2000%3A27%20UTC%20through%2008%2F01%2C%2000%3A41%20UTC%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3EWe%20understand%20that%20customers%20rely%20on%20Azure%20Monitor%20as%20a%20critical%20service%20and%20apologize%20for%20any%20impact%20this%20incident%20caused.%3C%2FDIV%3E%3CBR%20%2F%3E-Sindhu%3CBR%20%2F%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22%22%3E%3CDIV%20style%3D%22%22%3EUpdate%3A%20Wednesday%2C%2031%20July%202019%2023%3A52%20UTC%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3EThere%20was%20an%20API%20contract%20issue%20between%20our%20dependent%20services%20which%20caused%20metric%20alert%20rule%20creation%20failure%20for%20Network%20Resource%20Type.%26nbsp%3B%20To%20address%20this%20issue%20the%20engineers%20deployed%20a%20hotfix.%26nbsp%3B%20Some%20customers%20globally%20may%20experience%20failures%20while%20performing%20CRUD%20operations(create%2C%20modify%20or%20delete)%20for%20alerting%20when%20network%20metric%20alerts%20are%20included%20in%20the%20configuration.%20Existing%20alerting%20will%20work%20as%20expected.%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3EWork%20Around%3A%20none%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3ENext%20Update%3A%20Before%2008%2F01%2002%3A00%20UTC%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E-Sindhu%3C%2FDIV%3E%3C%2FDIV%3E%3CHR%20style%3D%22border-top-color%3Alightgray%22%20%2F%3E%3CDIV%20style%3D%22font-size%3A14px%3B%22%3E%3CDIV%20style%3D%22%22%3E%3CDIV%20style%3D%22%22%3E%3CDIV%20style%3D%22%22%3E%3CDIV%20style%3D%22%22%3EUpdate%3A%20Wednesday%2C%2031%20July%202019%2022%3A07%20UTC%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3EWe%20continue%20to%20investigate%20issues%20within%20Azure%20Monitoring%20and%20are%20actively%20investigating.%20Some%20customers%20globally%20may%20experience%20failures%20while%20performing%20CRUD%20operations(create%2C%20modify%20or%20delete)%20for%20alerting%20when%20network%20metric%20alerts%20are%20included%20in%20the%20configuration.%20Existing%20alerting%20would%20work%20as%20expected%3B%20impact%20is%20limited%20to%20CRUD.%26nbsp%3B%20We%20are%20preparing%20a%20hot%20fix.%20Once%20deployed%20the%20issue%20should%20be%20mitigated.%26nbsp%3B%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3EWork%20Around%3A%20None%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3ENext%20Update%3A%20Before%2008%2F01%2000%3A07%20UTC%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E-Ian%20Cairns%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3CHR%20style%3D%22font-size%3A%2014px%3B%20border-top-color%3A%20lightgray%3B%22%20%2F%3E%3CDIV%20style%3D%22%22%3E%3CDIV%20style%3D%22%22%3E%3CDIV%20style%3D%22%22%3E%3CDIV%20style%3D%22%22%3E%3CDIV%20style%3D%22%22%3E%3CDIV%20style%3D%22%22%3E%3CSPAN%20style%3D%22font-family%3A%20%22%20segoe%3D%22%22%20ui%3D%22%22%3EInitial%20Update%3A%20Wednesday%2C%2031%20July%202019%2020%3A00%20UTC%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E%3CSPAN%20style%3D%22font-family%3A%20%22%20segoe%3D%22%22%20ui%3D%22%22%3E%3CBR%20%2F%3E%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E%3CSPAN%20style%3D%22font-family%3A%20%22%20segoe%3D%22%22%20ui%3D%22%22%3EWe%20are%20aware%20of%20issues%20within%20Azure%20Monitoring%20and%20are%20actively%20investigating.%20Some%20customers%20globally%20may%20experience%20failures%20while%20performing%20CRUD%20operations(create%2C%20modify%20or%20delete)%20for%20alerting%20when%20network%20metric%20alerts%20are%20included%20in%20the%20configuration.%20Existing%20alerting%20will%20work%20as%20expected.%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E%3CSPAN%20style%3D%22font-family%3A%20%22%20segoe%3D%22%22%20ui%3D%22%22%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E%3CSPAN%20style%3D%22font-family%3A%20%22%20segoe%3D%22%22%20ui%3D%22%22%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E%3CSPAN%20style%3D%22font-family%3A%20%22%20segoe%3D%22%22%20ui%3D%22%22%3EWork%20Around%3A%20None%26nbsp%3B%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E%3CSPAN%20style%3D%22font-family%3A%20%22%20segoe%3D%22%22%20ui%3D%22%22%3ENext%20Update%3A%20Before%2007%2F31%2022%3A00%20UTC%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E%3CSPAN%20style%3D%22font-family%3A%20%22%20segoe%3D%22%22%20ui%3D%22%22%3EWe%20are%20working%20hard%20to%20resolve%20this%20issue%20and%20apologize%20for%20any%20inconvenience.%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E%3CSPAN%20style%3D%22font-family%3A%20%22%20segoe%3D%22%22%20ui%3D%22%22%3E%3CBR%20%2F%3E%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%20style%3D%22%22%3E%3CSPAN%20style%3D%22font-family%3A%20%22%20segoe%3D%22%22%20ui%3D%22%22%3E-Ian%20Cairns%3C%2FSPAN%3E%3C%2FDIV%3E%3C%2FDIV%3E%3CDIV%20style%3D%22font-family%3A%20%22%20segoe%3D%22%22%20ui%3D%22%22%3E%3CBR%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3CHR%20style%3D%22font-size%3A%2014px%3B%20border-top-color%3A%20lightgray%3B%22%20%2F%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-782330%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EApplication%20Insights%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Final Update: Thursday, 01 August 2019 00:27 UTC

We've confirmed that all systems are back to normal with no customer impact as of 01/08, 00:27 UTC. Our logs show the incident started on 07/27, 00:16 UTC and that during the duration of 5 days that it took to resolve the issue some customers globally may have experienced failures while performing CRUD operations(create, modify or delete) for alerting when network metric alerts are included in the configuration. Existing alerting will work as expected.

Root Cause: The failure was due to API contract issue between our dependent services which caused metric alert rule creation failure for Network Resource Type.

Incident Timeline: 5 days  - 07/27, 00:27 UTC through 08/01, 00:41 UTC

We understand that customers rely on Azure Monitor as a critical service and apologize for any impact this incident caused.

-Sindhu

Update: Wednesday, 31 July 2019 23:52 UTC

There was an API contract issue between our dependent services which caused metric alert rule creation failure for Network Resource Type.  To address this issue the engineers deployed a hotfix.  Some customers globally may experience failures while performing CRUD operations(create, modify or delete) for alerting when network metric alerts are included in the configuration. Existing alerting will work as expected.

Work Around: none
Next Update: Before 08/01 02:00 UTC

-Sindhu

Update: Wednesday, 31 July 2019 22:07 UTC

We continue to investigate issues within Azure Monitoring and are actively investigating. Some customers globally may experience failures while performing CRUD operations(create, modify or delete) for alerting when network metric alerts are included in the configuration. Existing alerting would work as expected; impact is limited to CRUD.  We are preparing a hot fix. Once deployed the issue should be mitigated. 

Work Around: None
Next Update: Before 08/01 00:07 UTC

-Ian Cairns

Initial Update: Wednesday, 31 July 2019 20:00 UTC

We are aware of issues within Azure Monitoring and are actively investigating. Some customers globally may experience failures while performing CRUD operations(create, modify or delete) for alerting when network metric alerts are included in the configuration. Existing alerting will work as expected.
 
 
Work Around: None 
Next Update: Before 07/31 22:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.

-Ian Cairns