Overview
We would like to extend our apologies to all our customers that have been impacted by the incident caused by an incompatibility between Azure PowerShell and PowerShell 7.3.
This article reviews the root cause of the issue, how long it took to implement a solution and exposes the corrective actions that we are taking.
Teams involved:
- Customer experience team
- PowerShell team
- Compute SDK team
- Azure PowerShell team
Summary of GitHub issues:
What happened?
Starting July 17, community customers have reported problems to our customer experience team. The customer originally ran the Azure PowerShell cmdlet Get-AzVM on PowerShell 7.2.5. It worked fine, but an error was reported after Visual Studio Code (VS Code) recommended upgrading to PowerShell 7.3.0.5 . The exception in the logs showed that it might be related to the AutoMapper package. After the team located it, given the mitigation solution, the customer rolled back to the previous version of PowerShell and closed the issue on GitHub.
When several customers reported this issue on November 9 and it appeared after PowerShell was upgraded to version 7.3.0.5, we realized that it was a critical problem. On November 16, the team released a new version to completely resolve the issue.
The following describes with more details the discovery and triage of the issue, and time trajectory to the solution.
Timeline
- Nov 9, 2022 – PowerShell 7.3 Generally Available.
- 8 PM, Nov 9, 3 users reported issues and added comments to one closed GitHub issue #18721. The user cannot use Az.Compute cmdlets when VS Code had updated PowerShell to 7.3.
- 9 PM, Nov 9, Azure PowerShell team acknowledged the issues with P0 severity and pinned the issue for customer awareness.
- 10 PM, Nov 9, Azure PowerShell team found Az.Network cmdlets had the same problem. We recommended the user to use PowerShell 7.2 or lower version as a workaround.
- 8 AM, Nov 10, Azure PowerShell team identified eight impacted modules with high usage.
- 10 AM, Nov 11, Azure PowerShell team provided the engineering build of Az.Compute with AutoMapper 8.1.1. The community validated the fix in Az.Compute.
- First mitigation: 4 PM, Nov 12, Azure PowerShell team provided engineering build of all eight impacted modules based on AutoMapper 8.1.1.
- 1 PM, Nov 14, users reported Az.Network still had the problem with another resource type Network Interface.
- 12 AM, Nov 15, Azure PowerShell team confirmed breaking changes between AutoMapper 6.2.2 and 8.1.1. It cannot be fixed based on AutoMapper 8.1.1.
- Final mitigation: 3 PM, Nov 15, Azure PowerShell team provided a new version of the engineering build based on forked AutoMapper 6.2.2. The official version was planned to be published on Nov 18 if there was no negative feedback from the community.
- 5 AM, Nov 16, Azure DevOps hosted pipelines and GitHub Actions upgraded PowerShell to 7.3 increasing the impact of the issue.
- 11 PM, Nov 17, Azure PowerShell team published a new version of Az with eight impacted modules.
Initial signal
- 9 PM, 17 Jul, the customer support team received an email from the internal service feedback problem “Subject: Microsoft.Compute | Get-AzVM fails with AutoMapper exception in PowerShell 7.3.0.5 + version”; The issue brief is as follows: Customer is running the PS cmdlet Get-AzVM and get the below error after installing PS preview 7.3.0.5 - these cmdlets were working fine on PS preview 7.3.0.4. Also, they work fine on PS 7.2.5. Moreover, from the log, the AutoMapper module throws the exception.
- 01 Aug, After the preliminary triage, the problem occurred in the compute module and was forwarded to the compute team to continue checking the issue.
- The problem can be reproduced, but the root cause is not identified:
- 09 Aug: Compute team focused on this issue. Compute team narrowed the issue down to AutoMapper.
- 1 AM,13 Aug: PowerShell team involved in the investigation.
- 15 Aug - Suggestion made to change the implementation of the Get-AzVM cmdlet.
- 17 Aug - Customer rolled back to the previous version of PowerShell and closed the issue on GitHub, and investigations stopped.
Impact
This issue impacted all users using those 9 modules on PowerShell 7.3. Users would experience the problem when using Azure PowerShell cmdlets in VS Code, Azure DevOps, GitHub Actions, or in their local environments.
Impacted modules:
- Az.Aks
- Az.ApiManagement
- Az.Compute
- Az.Maintenance
- Az.Monitor
- Az.Network
- Az.Peering
- Az.RecoveryServices
- Az.Resources
Root causes
The problem is due to a compatibility issue between AutoMapper and .NET 7.
AutoMapper is an external component used to parse the object type defined in SDK to PSCustomObject. The approach helps modules to mitigate breaking changes introduced by the service. The nine Azure PowerShell modules mentioned above use this approach.
.NET 7 has corrected the behavior of private method MakeGenericMethod() in System.Reflection. This change causes errors in AutoMapper when arguments do not meet any constraints defined on the generic parameters. Due to PowerShell 7.3 depending on .NET 7, users encounter this problem in those 9 modules on PowerShell 7.3 or lower versions of PowerShell on .NET 7.
Mitigation and resolution
The .NET team has no plan to revert this change and the latest AutoMapper version does not support .NET Standard 2.0. After consulting the AutoMapper author, the Azure PowerShell team decided to implement the fix in a fork AutoMapper 6.2.2 used in Azure PowerShell modules.
A patched versions of the 9 modules impacted and a new version of Az has been published. Users need to upgrade Az to 9.1.1 or higher if they want to use those modules on PowerShell 7.3 or .NET 7.
Lessons learned.
- Missing test with PowerShell preview version. Previews of PowerShell were not covered in the daily test of Azure PowerShell missing an early the detection of the issue with PowerShell 7.3.
- Optimize communication. Azure PowerShell team relies on verbal notification from the PowerShell team. This mechanism needs to be more sustainable, and a long-term communication solution must be established.
- Ability to identify critical issues. The first issue was reported five months ago based on PowerShell 7.3.0.5-preview. The severity of the early issue was underestimated.
- Increasing risk of legacy libraries. AutoMapper 6.1.1 was released five years ago. In addition, AutoMapper ended .NET standard 2.0 support since the 6. was released on Jan 5. Because Azure PowerShell is supported on Windows PowerShell support, using the latest version of AutoMapper was not an option. The external dependencies of Azure PowerShell create a risk that will increase over time.
Corrective actions
Short term
- Work with respective module owners to prevent new usage of AutoMapper in Azure PowerShell.
- Forked AutoMapper 6.2.2 to Azure PowerShell repo to remove our dependency on the external component.
- Azure PowerShell is removing the dependency on AutoMapper in the following modules.
- AKS (Azure Kubernetes Service)
- Monitor
- Resources
Long term
- Technical: Our current solution is to use forked and customized version of AutoMapper 6.2.2. We will work with owners of modules using AutoMapper towards reducing drastically or removing this dependency.
- Quality and verification: We are integrating the latest preview version of PowerShell into Azure PowerShell daily and smoke tests.
- Process Improvements:
- We will establish a process to review critical issues with the customer experience team and the PowerShell team is recommended.
- Create awareness amongst the support teams how to identify issues with potentially significant impact.
- Improve our notification mechanism to Azure services where Azure PowerShell is used to be able to bloc deployment in case of critical issues.
The Azure PowerShell team
