Starting July 17, community customers have reported problems to our customer experience team. The customer originally ran the Azure PowerShell cmdlet Get-AzVM on PowerShell 7.2.5. It worked fine, but an error was reported after Visual Studio Code (VS Code) recommended upgrading to PowerShell 184.108.40.206 . The exception in the logs showed that it might be related to the AutoMapper package. After the team located it, given the mitigation solution, the customer rolled back to the previous version of PowerShell and closed the issue on GitHub.
When several customers reported this issue on November 9 and it appeared after PowerShell was upgraded to version 220.127.116.11, we realized that it was a critical problem. On November 16, the team released a new version to completely resolve the issue.
The following describes with more details the discovery and triage of the issue, and time trajectory to the solution.
Nov 9, 2022 – PowerShell 7.3 Generally Available.
8 PM, Nov 9, 3 users reported issues and added comments to one closed GitHub issue #18721. The user cannot use Az.Compute cmdlets when VS Code had updated PowerShell to 7.3.
9 PM, Nov 9, Azure PowerShell team acknowledged the issues with P0 severity and pinned the issue for customer awareness.
10 PM, Nov 9, Azure PowerShell team found Az.Network cmdlets had the same problem. We recommended the user to use PowerShell 7.2 or lower version as a workaround.
8 AM, Nov 10, Azure PowerShell team identified eight impacted modules with high usage.
10 AM, Nov 11, Azure PowerShell team provided the engineering build of Az.Compute with AutoMapper 8.1.1. The community validated the fix in Az.Compute.
First mitigation: 4 PM, Nov 12, Azure PowerShell team provided engineering build of all eight impacted modules based on AutoMapper 8.1.1.
1 PM, Nov 14, users reported Az.Network still had the problem with another resource type Network Interface.
12 AM, Nov 15, Azure PowerShell team confirmed breaking changes between AutoMapper 6.2.2 and 8.1.1. It cannot be fixed based on AutoMapper 8.1.1.
Final mitigation: 3 PM, Nov 15, Azure PowerShell team provided a new version of the engineering build based on forked AutoMapper 6.2.2. The official version was planned to be published on Nov 18 if there was no negative feedback from the community.
5 AM, Nov 16, Azure DevOps hosted pipelines and GitHub Actions upgraded PowerShell to 7.3 increasing the impact of the issue.
11 PM, Nov 17, Azure PowerShell team published a new version of Az with eight impacted modules.
9 PM, 17 Jul, the customer support team received an email from the internal service feedback problem “Subject: Microsoft.Compute | Get-AzVM fails with AutoMapper exception in PowerShell 18.104.22.168 + version”; The issue brief is as follows: Customer is running the PS cmdlet Get-AzVM and get the below error after installing PS preview 22.214.171.124 - these cmdlets were working fine on PS preview 126.96.36.199. Also, they work fine on PS 7.2.5. Moreover, from the log, the AutoMapper module throws the exception.
01 Aug, After the preliminary triage, the problem occurred in the compute module and was forwarded to the compute team to continue checking the issue.
The problem can be reproduced, but the root cause is not identified:
09 Aug: Compute team focused on this issue. Compute team narrowed the issue down to AutoMapper.
1 AM,13 Aug: PowerShell team involved in the investigation.
15 Aug - Suggestion made to change the implementation of the Get-AzVM cmdlet.
17 Aug - Customer rolled back to the previous version of PowerShell and closed the issue on GitHub, and investigations stopped.
This issue impacted all users using those 9 modules on PowerShell 7.3. Users would experience the problem when using Azure PowerShell cmdlets in VS Code, Azure DevOps, GitHub Actions, or in their local environments.
The problem is due to a compatibility issue between AutoMapper and .NET 7.
AutoMapper is an external component used to parse the object type defined in SDK to PSCustomObject. The approach helps modules to mitigate breaking changes introduced by the service. The nine Azure PowerShell modules mentioned above use this approach.
.NET 7 has corrected the behavior of private method MakeGenericMethod() in System.Reflection. This change causes errors in AutoMapper when arguments do not meet any constraints defined on the generic parameters. Due to PowerShell 7.3 depending on .NET 7, users encounter this problem in those 9 modules on PowerShell 7.3 or lower versions of PowerShell on .NET 7.
Mitigation and resolution
The .NET team has no plan to revert this change and the latest AutoMapper version does not support .NET Standard 2.0. After consulting the AutoMapper author, the Azure PowerShell team decided to implement the fix in a fork AutoMapper 6.2.2 used in Azure PowerShell modules.
A patched versions of the 9 modules impacted and a new version of Az has been published. Users need to upgrade Az to 9.1.1 or higher if they want to use those modules on PowerShell 7.3 or .NET 7.
Missing test with PowerShell preview version. Previews of PowerShell were not covered in the daily test of Azure PowerShell missing an early the detection of the issue with PowerShell 7.3.
Optimize communication. Azure PowerShell team relies on verbal notification from the PowerShell team. This mechanism needs to be more sustainable, and a long-term communication solution must be established.
Ability to identify critical issues. The first issue was reported five months ago based on PowerShell 188.8.131.52-preview. The severity of the early issue was underestimated.
Increasing risk of legacy libraries. AutoMapper 6.1.1 was released five years ago. In addition, AutoMapper ended .NET standard 2.0 support since the 6. was released on Jan 5. Because Azure PowerShell is supported on Windows PowerShell support, using the latest version of AutoMapper was not an option. The external dependencies of Azure PowerShell create a risk that will increase over time.
Work with respective module owners to prevent new usage of AutoMapper in Azure PowerShell.
Forked AutoMapper 6.2.2 to Azure PowerShell repo to remove our dependency on the external component.
Azure PowerShell is removing the dependency on AutoMapper in the following modules.
AKS (Azure Kubernetes Service)
Technical: Our current solution is to use forked and customized version of AutoMapper 6.2.2. We will work with owners of modules using AutoMapper towards reducing drastically or removing this dependency.
Quality and verification: We are integrating the latest preview version of PowerShell into Azure PowerShell daily and smoke tests.
We will establish a process to review critical issues with the customer experience team and the PowerShell team is recommended.
Create awareness amongst the support teams how to identify issues with potentially significant impact.
Improve our notification mechanism to Azure services where Azure PowerShell is used to be able to bloc deployment in case of critical issues.