Embracing AKS built-in upgrade features and exploring custom solutions

Upgrading your AKS clusters in production is made simple with Microsoft’s robust, automated upgrade and update features. The official AKS upgrade process seamlessly handles surge nodes, optimizes Pod Disruption Budgets (PDBs), manages node updates, and performs comprehensive compatibility checks—all ensuring a smooth, low-downtime experience with minimal manual intervention.

Official AKS upgrade and update features 

Microsoft has built a rich set of features into AKS to simplify the upgrade process: 

• Automated cluster upgrades: AKS provides an automated upgrade process via the az aks upgrade command. This process manages surge nodes for availability, applies necessary health checks, and ensures minimal disruption during the upgrade. 

• Scheduled and auto-upgrades: With scheduled upgrades, you can define maintenance windows for cluster updates. The auto-upgrade feature (when enabled) automatically updates clusters, ensuring they remain under support without manual intervention. 

• Node image upgrades: The AKS Node Image Upgrade process automatically updates the underlying node images, reducing the risk of security vulnerabilities and compatibility issues. 

• Fleet orchestration for multi-cluster management: For organizations managing multiple clusters, Kubernetes Fleet Update Orchestration provides a centralized way to coordinate upgrades and updates across your entire fleet. 

These features are robust and continuously evolving, ensuring your production clusters are maintained with industry’s best practices.

Why consider a custom upgrade approach? 

For most users, leveraging the builtin AKS upgrade capabilities is the best way to maintain and update clusters.  However, some users desire complete control over every step of the process. If you have unique requirements—for instance, if you prefer to manually trigger upgrades ondemand rather than using scheduled upgrades, or if you need to integrate custom health checks and rollback logic—the custom CLI script presented in this post may be of interest.  

 
Disclaimer: This experimental proof-of-concept custom CLI solution is provided as-is and is not an official Microsoft solution. It hasn’t been tested on every supported configuration and is not production ready. Use it at your own risk and discretion. 

By exploring this custom approach, you may gain additional control over the upgrade process. Nevertheless, we strongly encourage most users to leverage the robust, builtin features provided by AKS. 

The custom CLI script for AKS upgrades 

For users interested in a more granular approach, this custom CLI script automates many aspects of the upgrade process. The script: 

• Displays available information: It lists your resource groups and AKS clusters (with resource group, cluster name, and location) so you can easily obtain the required parameters. 

• Dynamic credential download: The script automatically downloads your cluster credentials based on the Resource Group and Cluster Name you provide. 

• Retrieves the current version and allowed upgrade paths: It displays your current Kubernetes version and uses an interactive menu to show available upgrade targets, clearly marking allowed options. 

• Performs pre-upgrade health checks: The script checks node readiness, PDBs, failed pods, and even includes a placeholder for surge capacity. 

• Ensures compatibility checks: It reminds you to verify that your workloads are compatible with the new version before proceeding. 

• Initiates the upgrade process: Once you confirm, the script triggers the upgrade using the az aks upgrade command. 

• Validates post-upgrade health: After upgrading, the script verifies application health and provides a simulated rollback option if issues are detected.

AKS upgrade script:

#!/bin/bash
###############################################################################
# Enhanced AKS Upgrade Script with Health Validation & Rollback
#
# Prerequisites are handled within this script:
#   1. List your subscriptions to get the subscription ID:
#      az account list --output table
#   2. Set your subscription:
#      az account set --subscription 00000000-0000-0000-0000-000000000000
#   3. List your resource groups:
#      az group list --output table
#   4. List your AKS clusters (ResourceGroup, Cluster Name, Location):
#      az aks list --output table
#   5. The script downloads your cluster credentials dynamically.
#
# This script retrieves the current Kubernetes version for an AKS cluster,
# shows allowed upgrade paths and highlights allowed options, performs 
# pre-upgrade health and compatibility checks, initiates the upgrade process, 
# validates application health post-upgrade, and offers a simulated rollback if issues 
# are detected.
#
# NOTE: AKS does not officially support downgrades. The rollback here simulates
#       a recovery by re-upgrading to the previous version.
###############################################################################

#---------------------------
# Display Available Resource Groups and AKS Clusters
#---------------------------
echo "------ Available Resource Groups ------"
az group list --output table
echo ""
echo "------ Available AKS Clusters (ResourceGroup, Cluster Name, Location) ------"
az aks list --query "[].{ResourceGroup: resourceGroup, ClusterName: name, Location: location}" -o table
echo ""
echo "Please note the Resource Group, Cluster Name, and Location for your AKS cluster."
echo "----------------------------------------------------------------------"
echo ""

#---------------------------
# Helper Functions
#---------------------------

# Pre-upgrade health checks (nodes, PDBs, failed pods, surge capacity)
perform_pre_upgrade_checks() {
    echo ""
    echo "--------------------------------------------"
    echo "Performing Pre-Upgrade Health Checks"
    echo "--------------------------------------------"
    
    echo "1. Checking node status..."
    kubectl get nodes

    echo ""
    echo "2. Checking for any NotReady nodes..."
    NOTREADY=$(kubectl get nodes | grep NotReady)
    if [ ! -z "$NOTREADY" ]; then
        echo "WARNING: Some nodes are not ready. Please investigate before upgrading."
    else
        echo "All nodes are Ready."
    fi

    echo ""
    echo "3. Checking Pod Disruption Budgets (PDBs)..."
    kubectl get pdb --all-namespaces

    echo ""
    echo "4. Checking for pods in a failed state (e.g., CrashLoopBackOff, Error)..."
    kubectl get pods --all-namespaces | grep -E 'CrashLoopBackOff|Error' || echo "No pods in error state found."

    echo ""
    echo "5. Checking for surge nodes / additional capacity (placeholder)..."
    echo "   (Ensure your node pool autoscaler or surge capacity is configured properly.)"
    
    echo ""
    echo "Pre-upgrade health checks completed. Please review the output above."
    read -p "Do you want to continue with the upgrade? (y/N): " CHECK_CONFIRM
    if [[ ! "$CHECK_CONFIRM" =~ ^[Yy]$ ]]; then
        echo "Upgrade cancelled based on pre-upgrade health checks."
        exit 1
    fi
}

# Compatibility check reminder
perform_compatibility_checks() {
    echo ""
    echo "--------------------------------------------"
    echo "Performing Compatibility Checks"
    echo "--------------------------------------------"
    echo "NOTE: Ensure that all critical workloads, custom resources, and third-party"
    echo "      integrations are compatible with the new Kubernetes version."
    echo "      Review release notes and documentation for any breaking changes."
    echo ""
    read -p "Have you verified workload compatibility? (y/N): " COMP_CONFIRM
    if [[ ! "$COMP_CONFIRM" =~ ^[Yy]$ ]]; then
        echo "Upgrade cancelled. Please verify compatibility and try again."
        exit 1
    fi
}

# Post-upgrade health checks (applications, deployments, pods)
perform_post_upgrade_checks() {
    echo ""
    echo "--------------------------------------------"
    echo "Performing Post-Upgrade Health Checks"
    echo "--------------------------------------------"
    echo "Checking deployments status..."
    kubectl get deployments --all-namespaces

    echo ""
    echo "Checking pods status..."
    kubectl get pods --all-namespaces

    echo ""
    echo "Please review the output for any errors or issues with your applications."
    read -p "Do all applications appear healthy? (y/N): " POST_CHECK_CONFIRM
    if [[ ! "$POST_CHECK_CONFIRM" =~ ^[Yy]$ ]]; then
        return 1
    fi
    return 0
}

# Attempt rollback to previous version (simulation)
attempt_rollback() {
    echo ""
    echo "--------------------------------------------"
    echo "Attempting Rollback"
    echo "--------------------------------------------"
    read -p "Rollback to the previous version ($CURRENT_VERSION) ? (y/N): " ROLLBACK_CONFIRM
    if [[ "$ROLLBACK_CONFIRM" =~ ^[Yy]$ ]]; then
        echo "Initiating rollback to version $CURRENT_VERSION..."
        az aks upgrade --resource-group "$RESOURCE_GROUP" --name "$CLUSTER_NAME" --kubernetes-version "$CURRENT_VERSION" --yes
        if [ $? -eq 0 ]; then
            echo "Rollback executed successfully."
        else
            echo "Rollback failed. Please check the error messages above and consider manual recovery."
            exit 1
        fi
    else
        echo "Rollback aborted. Please perform manual recovery if necessary."
        exit 1
    fi
}

#---------------------------
# Main Script
#---------------------------

# Prompt for input parameters
read -p "Enter the Resource Group: " RESOURCE_GROUP
read -p "Enter the AKS Cluster Name: " CLUSTER_NAME
read -p "Enter the AKS Region (e.g., eastus): " LOCATION

# Download cluster credentials dynamically
echo ""
echo "Downloading cluster credentials for '$CLUSTER_NAME' in resource group '$RESOURCE_GROUP'..."
az aks get-credentials --resource-group "$RESOURCE_GROUP" --name "$CLUSTER_NAME" --overwrite-existing

###############################################################################
# Step 1: Retrieve and Display the Current Kubernetes Version
###############################################################################
echo ""
echo "Fetching the current Kubernetes version for cluster '$CLUSTER_NAME' in '$RESOURCE_GROUP'..."
CURRENT_VERSION=$(az aks show --resource-group "$RESOURCE_GROUP" --name "$CLUSTER_NAME" --query "kubernetesVersion" -o tsv)
if [ -z "$CURRENT_VERSION" ]; then
    echo "ERROR: Failed to retrieve the current Kubernetes version. Please check your cluster details."
    exit 1
fi
echo "Current Kubernetes version: $CURRENT_VERSION"
echo ""

###############################################################################
# Step 2: Retrieve Allowed Upgrade Paths for the Cluster
###############################################################################
echo "Retrieving allowed upgrade paths for your cluster..."
UPGRADES_JSON=$(az aks get-upgrades --resource-group "$RESOURCE_GROUP" --name "$CLUSTER_NAME" -o json)
ALLOWED_UPGRADES=$(echo "$UPGRADES_JSON" | jq -r 'if .controlPlaneProfile.upgradeProfile.upgrades then (.controlPlaneProfile.upgradeProfile.upgrades | map(.kubernetesVersion) | join(" ")) else "" end')

# Fallback: if no allowed upgrades are determined and CURRENT_VERSION starts with "1.30"
if [ -z "$ALLOWED_UPGRADES" ]; then
    if [[ "$CURRENT_VERSION" =~ ^1\.30 ]]; then
        echo "WARNING: No allowed upgrade paths could be determined automatically."
        echo "Typically, if your cluster is running a version like 1.30.x (e.g., 1.30.10),"
        echo "you can only upgrade directly to a 1.31.x version."
        echo "For example, allowed upgrade targets might include: 1.31.6, 1.31.5, 1.31.4, 1.31.3, 1.31.2, or 1.31.1."
        ALLOWED_UPGRADES="1.31.6 1.31.5 1.31.4 1.31.3 1.31.2 1.31.1"
        ALLOWED_MAJOR_MINOR="1.31"
    else
        echo "WARNING: No allowed upgrade paths could be determined automatically. Proceed with caution."
        ALLOWED_MAJOR_MINOR=""
    fi
else
    # Extract unique major.minor values from allowed upgrades
    ALLOWED_MAJOR_MINOR=$(for ver in $ALLOWED_UPGRADES; do
        echo "$ver" | awk -F. '{print $1"."$2}';
    done | sort -u | tr '\n' ' ')
fi

if [ -n "$ALLOWED_MAJOR_MINOR" ]; then
    echo ""
    echo "Based on your current version ($CURRENT_VERSION), you can upgrade directly to versions with major.minor:"
    for mm in $ALLOWED_MAJOR_MINOR; do
        echo "  - $mm"
    done
    echo "Only versions matching these allowed major.minor values will be marked as [ALLOWED] below."
    echo "For more details, please see https://aka.ms/aks-supported-k8s-ver"
    echo ""
fi

###############################################################################
# Step 3: Fetch Available Kubernetes Versions
###############################################################################
echo "Fetching available Kubernetes versions in '$LOCATION'..."
VERSIONS_JSON=$(az aks get-versions --location "$LOCATION" -o json)
if [ $? -ne 0 ]; then
    echo "ERROR: Failed to fetch available versions. Please check your Azure CLI configuration."
    exit 1
fi

# Extract list of versions and preview flag from the "values" array
mapfile -t OPTIONS < <(echo "$VERSIONS_JSON" | jq -r '(.values // [])[] | "\(.version) \(.isPreview)"')

if [ ${#OPTIONS[@]} -eq 0 ]; then
    echo "ERROR: No versions found for location '$LOCATION'."
    echo "This might be due to the aks-preview extension altering the output."
    echo "If you don't need preview features, try removing the extension with: az extension remove --name aks-preview"
    exit 1
fi

###############################################################################
# Step 4: Build the Interactive Menu with Highlighted Allowed Options
###############################################################################
declare -a VERSION_LIST
declare -a LABELS

for entry in "${OPTIONS[@]}"; do
    # Extract version and preview flag
    VERSION=$(echo "$entry" | awk '{print $1}')
    IS_PREVIEW=$(echo "$entry" | awk '{print $2}')
    LABEL="$VERSION"
    if [ "$IS_PREVIEW" == "true" ]; then
        LABEL="$LABEL (Preview)"
    else
        LABEL="$LABEL (Stable)"
    fi
    
    # Highlight if allowed (by comparing major.minor)
    if [ -n "$ALLOWED_MAJOR_MINOR" ]; then
        AVAILABLE_MM=$(echo "$VERSION" | awk -F. '{print $1"."$2}')
        for allowed in $ALLOWED_MAJOR_MINOR; do
            if [ "$AVAILABLE_MM" == "$allowed" ]; then
                LABEL="$LABEL [ALLOWED]"
                break
            fi
        done
    fi

    VERSION_LIST+=("$VERSION")
    LABELS+=("$LABEL")
done

echo ""
echo "Select the Kubernetes version to upgrade to:"
PS3="Enter your choice (or type 'q' to quit): "
select opt in "${LABELS[@]}"; do
    if [[ "$REPLY" == "q" ]]; then
        echo "Exiting..."
        exit 0
    fi
    if [ -z "$opt" ]; then
        echo "Invalid selection. Please try again."
    else
        TARGET_VERSION=${VERSION_LIST[$((REPLY-1))]}
        echo "You selected: $opt"
        break
    fi
done

###############################################################################
# Step 5: Validate the Selected Target Version
###############################################################################
if [ -n "$ALLOWED_MAJOR_MINOR" ]; then
    AVAILABLE_MM=$(echo "$TARGET_VERSION" | awk -F. '{print $1"."$2}')
    ALLOWED_MATCH=0
    for allowed in $ALLOWED_MAJOR_MINOR; do
        if [ "$AVAILABLE_MM" == "$allowed" ]; then
            ALLOWED_MATCH=1
            break
        fi
    done
    if [ $ALLOWED_MATCH -ne 1 ]; then
        echo ""
        echo "WARNING: Upgrading from $CURRENT_VERSION to $TARGET_VERSION is not allowed based on your cluster's upgrade policy."
        echo "Allowed upgrades from your current version are only for versions with major.minor:"
        for mm in $ALLOWED_MAJOR_MINOR; do
            echo "  - $mm"
        done
        echo "Please select one of these allowed versions."
        exit 1
    fi
fi

###############################################################################
# Step 6: Pre-Upgrade Health & Compatibility Checks
###############################################################################
perform_pre_upgrade_checks
perform_compatibility_checks

###############################################################################
# Step 7: Confirm and Execute the Upgrade
###############################################################################
echo ""
read -p "Proceed with upgrading '$CLUSTER_NAME' from version $CURRENT_VERSION to $TARGET_VERSION? (y/N): " CONFIRM
if [[ ! "$CONFIRM" =~ ^[Yy]$ ]]; then
    echo "Upgrade cancelled."
    exit 0
fi

echo ""
echo "Initiating upgrade to version $TARGET_VERSION..."
az aks upgrade --resource-group "$RESOURCE_GROUP" --name "$CLUSTER_NAME" --kubernetes-version "$TARGET_VERSION" --yes

if [ $? -eq 0 ]; then
    echo "Upgrade command executed successfully."
else
    echo "ERROR: Upgrade command failed. Please check the error messages above."
    exit 1
fi

###############################################################################
# Step 8: Post-Upgrade Health Checks & Rollback Option
# Rollback Mechanism Note: The rollback feature in this script is designed to simulate a recovery process by re-upgrading the cluster back to its previous version. 
# Please note that AKS does not officially support downgrades, so this rollback is not a true downgrade in the traditional sense. It is a best-effort approach that relies on having a known, working previous version and should only be used as a last resort. 
# Ensure that you have proper backups and recovery strategies in place before relying on this functionality.
###############################################################################
if ! perform_post_upgrade_checks; then
    echo ""
    echo "One or more post-upgrade health checks have failed."
    attempt_rollback
else
    echo ""
    echo "Post-upgrade health checks passed. Your applications appear healthy."
fi

echo ""
echo "Upgrade complete. Please continue monitoring your cluster and applications for any issues."

 
You can download the full script here: Custom CLI Script for AKS Upgrades  

Additional considerations 

Rollback mechanism note: The rollback feature in this script is designed to simulate a recovery process by re-upgrading the cluster to its previous version. Please note that AKS does not officially support downgrades, so this rollback is not a true downgrade in the traditional sense. It is a best-effort approach that relies on having a known, working previous version and should only be used as a last resort. Ensure you have proper backups and a comprehensive recovery strategy in place before relying on this functionality. 

Final thoughts 

Microsoft’s official AKS upgrade features are powerful and designed to simplify the process—from automated cluster and node image upgrades to orchestrated updates across multiple clusters using Fleet. For most users, these built-in capabilities offer the most reliable and supported approach. 

That said, if you’re a user with unique requirements, exploring a custom solution can provide granular control over every step of the process. This custom CLI script is provided as a proof-of-concept to inspire those who wish to tailor the upgrade process to their specific needs—but always remember, use it at your own risk. It is experimental, not production-ready, and is not fully supported by Microsoft.