When we redesigned the Azure CLI in 2016, our goal was to create a simple, easy to use, tool for managing Azure.  Since then, Azure has grown tremendously from supporting multiple clouds in various locations across the globe, on/off premises services, and scenario-specific technologies, such as Azure IoT and AI/ML tools. 

With over 2500 commands, tens of thousands of parameters, and new Azure features being developed daily, maintaining simplicity in Azure CLI is a challenge. In this post, we want to introduce you to three of our AI-powered features we’ve developed to ensure the Azure CLI remains easy to use, consistent, and “evergreen”.  These features include: generating up-to-date examples, enabling natural language search in command line, and assisting with failure recovery. We will share with you our learnings and journey below.

Generating up-to-date examples

Learning new software and APIs requires up to date documentation with multiple examples covering the main common scenarios. However, the fast pace of modern software causes documentation, especially examples, to quickly become stale.

We’ve addressed the problem of incomplete and stale documentation by developing an AI-powered platform that automatically generates and updates examples for our command line tools (Azure CLI and Azure PowerShell) ensuring

• Up-to-date examples: generate new examples with each new Azure CLI (and Azure PowerShell) release, ensuring the documentation is always up-to-date.
• Representative of actual usage: Unlike bare-bones examples usually found in documentation that only cover basic scenarios, our examples are based on actual usage patterns and therefore represent how current users use the software in practice.
• Informed by our educational content: Our platform ingests over 14,000 pages of blogs, tutorials, and samples to ensure consistency (naming, sample values, etc.) across our learning resources. 

Figure 1. Just like regular examples, our auto-generated examples are accessible through the command line by typing --help/-h in front of the command name. In this Figure, the user has called help on ‘az keyvault update’.

Our examples are accessible through the command line help by typing the command name followed by “--help” or “-h” in the command line (Figure 1). Additionally, examples are published to our online reference docs (Figure 2). 

Figure 2. Screenshot of help content and examples as shown in reference docs on the web.

To measure the effectiveness of our pipeline, we examined the coverage and quality of our generated examples. We observed the examples written by software owners (human generated examples) cover 55% of the functionality and features supported by Azure CLI, while our AI-generated examples cover 100% of used features and functionality. This means that algorithmically we can achieve a scale that cannot be achieved through manual writing of examples. Not only do AI-generated examples cover more commands, but they also cover more service functionality. For example, while human-generated examples on average only cover 20% of command parameters, AI-generated ones cover 32%.

Enabling natural language search in command line  

Finding the right command can be difficult, especially with multiple Azure offerings for popular concepts or scenarios. The Azure CLI originally shipped with an `az find` command powered by plain text search, allowing users to search through the 100s of commands for their scenario. However, the plain text search did not provide any recommendations or context on how to pick the best service. 

Figure 3. Our natural language search, finds the most relevant examples in our knowledgebase.

In early 2019, we released an AI based version of ‘az find’, augmented with natural language search features powered by the Azure Search Service (Figure 3). In addition, the new ‘az find’ also supplies command and command group level examples allowing users to easily find the most popular features and services (Figure 4). While our in-tool AI-powered examples are updated with each release, the ’az find’ command supplies the most up-to-date recommendations. 

Figure 4. For each command group, the most common commands used within that command group are shown.

Assisting in failure recovery

Taking the right action to recover from a failure can be challenging and exacerbated by an ever-changing platform where new functionalities and features are added frequently. Our cloud developers and in particular novice developers, may struggle with finding the right commands, parameters, or values and further struggle with cryptic error messages. Guiding developers through examples can help alleviate some of their frustrations. With the goal of helping such users, we present a command line failure recovery recommender system for Azure CLI. Our failure recovery system is an AI-based system that learns from other users’ past errors and creates recovery patterns from these errors.

An Example

Let’s consider a new Azure developer who has created a storage account before and now is using Azure CLI to quickly create another account. The user types the following command.

The previous Azure CLI versions would have shown the user an error saying ‘create’ is not in ‘az storage’ command group and referenced in editor help or generic online docs for help. However, with our failure recovery recommender system, the new Azure CLI will recommend an example command that the user is most likely to run.

How does it work?

Our failure recovery system uses a conditional probability model. This model is built on the assumption that scripts accomplish tasks and specific tasks require specific commands in a certain order. Within each script, if a command fails there should be common patterns on how the next successful command remedies that failure. Data analysis demonstrates that given one or more commands in a script significantly narrows the probability of what command(s) follow these in that script. For instance, the chance of a user running “az storage account create” is 0.01%; when proceeded by ‘az storage create’ which is a failure the chance increases to 49%.

How do we serve the recommendations?

The failure recommendation service is hosted by Azure App Service. Scalability and performance are critical for the service so that we can respond to requests quickly in the order of milliseconds. When the system is under heavy load, we leverage the autoscaling capabilities in Azure to quickly scale up and keep the service performant. The service is deployed into different regions worldwide, which is aligned with Azure CLI users distribution to make sure every user’s request can be served from a region within the shortest amount of time.

Help us improve with your feedback

Help us make the tools that you love and use every day even better. You can provide feedback by creating an issue in Azure CLI Github page.