Advancing safe deployment with AIOps—introducing Gandalf

%3CLINGO-SUB%20id%3D%22lingo-sub-2507680%22%20slang%3D%22en-US%22%3EAdvancing%20safe%20deployment%20with%20AIOps%E2%80%94introducing%20Gandalf%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2507680%22%20slang%3D%22en-US%22%3E%3CP%3E%3CEM%3EWritten%20by%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20title%3D%22Mark%20Russinovich's%20blog%20archive%22%20href%3D%22https%3A%2F%2Fazure.microsoft.com%2Fen-us%2Fblog%2Fauthor%2Fmarkruss%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EMark%20Russinovich%3C%2FA%3E%2C%20Chief%20Technology%20Officer%20and%20Technical%20Fellow%2C%20Microsoft%20Azure%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CBLOCKQUOTE%3E%0A%3CP%20class%3D%22x-hidden-focus%22%3E%E2%80%9CChanges%20to%20Azure%20services%20and%20the%20Azure%20platform%20itself%20are%20both%20inevitable%20and%20beneficial%2C%20to%20ensure%20continuous%20delivery%20of%20updates%2C%20new%20features%2C%20and%20security%20enhancements.%20However%2C%20change%20is%20also%20a%20primary%20cause%20of%20service%20regressions%20that%20can%20contribute%20towards%20reliability%20issues%E2%80%94for%20hyperscale%20cloud%20providers%2C%20indeed%20for%20any%20IT%20service%20provider.%20As%20such%2C%20it%20is%20critical%20to%20catch%20any%20such%20problems%20as%20early%20as%20possible%20during%20the%20development%20and%20deployment%20rollout%2C%20to%20minimize%20any%20impact%20on%20the%20customer%20experience.%20As%20part%20of%20our%20ongoing%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20title%3D%22Advancing%20Reliability%20blog%20series%20listing%22%20href%3D%22https%3A%2F%2Fwww.aka.ms%2FAdvancingReliability%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EAdvancing%20Reliability%3C%2FA%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Eblog%20series%2C%20today%20I%E2%80%99ve%20asked%20Principal%20Program%20Manager%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CSTRONG%3EJian%20Zhang%3C%2FSTRONG%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Efrom%20our%20AIOps%20team%20to%20introduce%20how%20we%E2%80%99re%20increasingly%20leveraging%20machine%20learning%20to%20de-risk%20these%20changes%2C%20ultimately%20to%20improve%20the%20reliability%20of%20Azure.%E2%80%9D%E2%80%94Mark%20Russinovich%2C%20CTO%2C%20Azure%3C%2FP%3E%0A%3C%2FBLOCKQUOTE%3E%0A%3CP%3E%3CEM%3EThis%20post%20includes%20contributions%20from%20Principal%20Data%20Scientists%20Ken%20Hsieh%20and%20Ze%20Li%2C%20Principal%20Data%20Scientist%20Manager%20Yingnong%20Dang%2C%20and%20Partner%20Group%20Software%20Engineering%20Manager%20Murali%20Chintalapati.%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20our%20earlier%20blog%20post%20%E2%80%9C%3CA%20title%3D%22Advancing%20safe%20deployment%20practices%22%20href%3D%22https%3A%2F%2Fazure.microsoft.com%2Fblog%2Fadvancing-safe-deployment-practices%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EAdvancing%20safe%20deployment%20practices%3C%2FA%3E%E2%80%9D%20Cristina%20del%20Amo%20Casado%20described%20how%20we%20release%20changes%20to%20production%2C%20for%20both%20code%20and%20configuration%20changes%2C%20across%20the%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CA%20title%3D%22azure.com%20%7C%20home%22%20href%3D%22https%3A%2F%2Fazure.microsoft.com%2Fen-us%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EAzure%3C%2FA%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Eplatform.%20The%20processes%20consist%20of%20delivering%20changes%20progressively%2C%20with%20phases%20that%20incorporate%20enough%20bake%20time%20to%20allow%20detection%20at%20a%20small%20scale%20for%20most%20regressions%20missed%20during%20testing.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20continuous%20monitoring%20of%20health%20metrics%20is%20a%20fundamental%20part%20of%20this%20process%2C%20and%20this%20is%20where%20AIOps%20plays%20a%20critical%20role%E2%80%94it%20allows%20the%20detection%20of%20anomalies%20to%20trigger%20alerts%20and%20the%20automation%20of%20correcting%20actions%20such%20as%20stopping%20the%20deployment%20or%20initiating%20rollbacks.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22%22%3EIn%20the%20post%20that%20follows%2C%20we%20introduce%20how%20AI%20and%20machine%20learning%20are%20used%20to%20empower%20DevOps%20engineers%2C%20monitor%20the%20Azure%20deployment%20process%20at%20scale%2C%20detect%20issues%20early%2C%20and%20make%20rollout%20or%20rollback%20decisions%20based%20on%20impact%20scope%20and%20severity.%3C%2FP%3E%0A%3CP%20class%3D%22%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20class%3D%22%22%3E%3CA%20title%3D%22Advancing%20safe%20deployment%20with%20AIOps%E2%80%94introducing%20Gandalf%22%20href%3D%22https%3A%2F%2Fazure.microsoft.com%2Fen-us%2Fblog%2Fadvancing-safe-deployment-with-aiops-introducing-gandalf%2F%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3E%3CSTRONG%3ERead%20the%20full%20article%3C%2FSTRONG%3E%3C%2FA%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E
Microsoft

Written by Mark Russinovich, Chief Technology Officer and Technical Fellow, Microsoft Azure

 

“Changes to Azure services and the Azure platform itself are both inevitable and beneficial, to ensure continuous delivery of updates, new features, and security enhancements. However, change is also a primary cause of service regressions that can contribute towards reliability issues—for hyperscale cloud providers, indeed for any IT service provider. As such, it is critical to catch any such problems as early as possible during the development and deployment rollout, to minimize any impact on the customer experience. As part of our ongoing Advancing Reliability blog series, today I’ve asked Principal Program Manager Jian Zhang from our AIOps team to introduce how we’re increasingly leveraging machine learning to de-risk these changes, ultimately to improve the reliability of Azure.”—Mark Russinovich, CTO, Azure

This post includes contributions from Principal Data Scientists Ken Hsieh and Ze Li, Principal Data Scientist Manager Yingnong Dang, and Partner Group Software Engineering Manager Murali Chintalapati.

 

In our earlier blog post “Advancing safe deployment practices” Cristina del Amo Casado described how we release changes to production, for both code and configuration changes, across the Azure platform. The processes consist of delivering changes progressively, with phases that incorporate enough bake time to allow detection at a small scale for most regressions missed during testing.

 

The continuous monitoring of health metrics is a fundamental part of this process, and this is where AIOps plays a critical role—it allows the detection of anomalies to trigger alerts and the automation of correcting actions such as stopping the deployment or initiating rollbacks.

 

In the post that follows, we introduce how AI and machine learning are used to empower DevOps engineers, monitor the Azure deployment process at scale, detect issues early, and make rollout or rollback decisions based on impact scope and severity.

 

Read the full article

0 Replies