Embracing AKS built-in upgrade features and exploring custom solutions
Upgrading your AKS clusters in production is made simple with Microsoft’s robust, automated upgrade and update features. The official AKS upgrade process seamlessly handles surge nodes, optimizes Pod Disruption Budgets (PDBs), manages node updates, and performs comprehensive compatibility checks—all ensuring a smooth, low-downtime experience with minimal manual intervention. Official AKS upgrade and update features Microsoft has built a rich set of features into AKS to simplify the upgrade process: Automated cluster upgrades: AKS provides an automated upgrade process via the az aks upgrade command. This process manages surge nodes for availability, applies necessary health checks, and ensures minimal disruption during the upgrade. Scheduled and auto-upgrades: With scheduled upgrades, you can define maintenance windows for cluster updates. The auto-upgrade feature (when enabled) automatically updates clusters, ensuring they remain under support without manual intervention. Node image upgrades: The AKS Node Image Upgrade process automatically updates the underlying node images, reducing the risk of security vulnerabilities and compatibility issues. Fleet orchestration for multi-cluster management: For organizations managing multiple clusters, Kubernetes Fleet Update Orchestration provides a centralized way to coordinate upgrades and updates across your entire fleet. These features are robust and continuously evolving, ensuring your production clusters are maintained with industry’s best practices. Why consider a custom upgrade approach? For most users, leveraging the builtin AKS upgrade capabilities is the best way to maintain and update clusters. However, some users desire complete control over every step of the process. If you have unique requirements—for instance, if you prefer to manually trigger upgrades ondemand rather than using scheduled upgrades, or if you need to integrate custom health checks and rollback logic—the custom CLI script presented in this post may be of interest. Disclaimer: This experimental proof-of-concept custom CLI solution is provided as-is and is not an official Microsoft solution. It hasn’t been tested on every supported configuration and is not production ready. Use it at your own risk and discretion. By exploring this custom approach, you may gain additional control over the upgrade process. Nevertheless, we strongly encourage most users to leverage the robust, builtin features provided by AKS. The custom CLI script for AKS upgrades For users interested in a more granular approach, this custom CLI script automates many aspects of the upgrade process. The script: Displays available information: It lists your resource groups and AKS clusters (with resource group, cluster name, and location) so you can easily obtain the required parameters. Dynamic credential download: The script automatically downloads your cluster credentials based on the Resource Group and Cluster Name you provide. Retrieves the current version and allowed upgrade paths: It displays your current Kubernetes version and uses an interactive menu to show available upgrade targets, clearly marking allowed options. Performs pre-upgrade health checks: The script checks node readiness, PDBs, failed pods, and even includes a placeholder for surge capacity. Ensures compatibility checks: It reminds you to verify that your workloads are compatible with the new version before proceeding. Initiates the upgrade process: Once you confirm, the script triggers the upgrade using the az aks upgrade command. Validates post-upgrade health: After upgrading, the script verifies application health and provides a simulated rollback option if issues are detected. AKS upgrade script: #!/bin/bash ############################################################################### # Enhanced AKS Upgrade Script with Health Validation & Rollback # # Prerequisites are handled within this script: # 1. List your subscriptions to get the subscription ID: # az account list --output table # 2. Set your subscription: # az account set --subscription 00000000-0000-0000-0000-000000000000 # 3. List your resource groups: # az group list --output table # 4. List your AKS clusters (ResourceGroup, Cluster Name, Location): # az aks list --output table # 5. The script downloads your cluster credentials dynamically. # # This script retrieves the current Kubernetes version for an AKS cluster, # shows allowed upgrade paths and highlights allowed options, performs # pre-upgrade health and compatibility checks, initiates the upgrade process, # validates application health post-upgrade, and offers a simulated rollback if issues # are detected. # # NOTE: AKS does not officially support downgrades. The rollback here simulates # a recovery by re-upgrading to the previous version. ############################################################################### #--------------------------- # Display Available Resource Groups and AKS Clusters #--------------------------- echo "------ Available Resource Groups ------" az group list --output table echo "" echo "------ Available AKS Clusters (ResourceGroup, Cluster Name, Location) ------" az aks list --query "[].{ResourceGroup: resourceGroup, ClusterName: name, Location: location}" -o table echo "" echo "Please note the Resource Group, Cluster Name, and Location for your AKS cluster." echo "----------------------------------------------------------------------" echo "" #--------------------------- # Helper Functions #--------------------------- # Pre-upgrade health checks (nodes, PDBs, failed pods, surge capacity) perform_pre_upgrade_checks() { echo "" echo "--------------------------------------------" echo "Performing Pre-Upgrade Health Checks" echo "--------------------------------------------" echo "1. Checking node status..." kubectl get nodes echo "" echo "2. Checking for any NotReady nodes..." NOTREADY=$(kubectl get nodes | grep NotReady) if [ ! -z "$NOTREADY" ]; then echo "WARNING: Some nodes are not ready. Please investigate before upgrading." else echo "All nodes are Ready." fi echo "" echo "3. Checking Pod Disruption Budgets (PDBs)..." kubectl get pdb --all-namespaces echo "" echo "4. Checking for pods in a failed state (e.g., CrashLoopBackOff, Error)..." kubectl get pods --all-namespaces | grep -E 'CrashLoopBackOff|Error' || echo "No pods in error state found." echo "" echo "5. Checking for surge nodes / additional capacity (placeholder)..." echo " (Ensure your node pool autoscaler or surge capacity is configured properly.)" echo "" echo "Pre-upgrade health checks completed. Please review the output above." read -p "Do you want to continue with the upgrade? (y/N): " CHECK_CONFIRM if [[ ! "$CHECK_CONFIRM" =~ ^[Yy]$ ]]; then echo "Upgrade cancelled based on pre-upgrade health checks." exit 1 fi } # Compatibility check reminder perform_compatibility_checks() { echo "" echo "--------------------------------------------" echo "Performing Compatibility Checks" echo "--------------------------------------------" echo "NOTE: Ensure that all critical workloads, custom resources, and third-party" echo " integrations are compatible with the new Kubernetes version." echo " Review release notes and documentation for any breaking changes." echo "" read -p "Have you verified workload compatibility? (y/N): " COMP_CONFIRM if [[ ! "$COMP_CONFIRM" =~ ^[Yy]$ ]]; then echo "Upgrade cancelled. Please verify compatibility and try again." exit 1 fi } # Post-upgrade health checks (applications, deployments, pods) perform_post_upgrade_checks() { echo "" echo "--------------------------------------------" echo "Performing Post-Upgrade Health Checks" echo "--------------------------------------------" echo "Checking deployments status..." kubectl get deployments --all-namespaces echo "" echo "Checking pods status..." kubectl get pods --all-namespaces echo "" echo "Please review the output for any errors or issues with your applications." read -p "Do all applications appear healthy? (y/N): " POST_CHECK_CONFIRM if [[ ! "$POST_CHECK_CONFIRM" =~ ^[Yy]$ ]]; then return 1 fi return 0 } # Attempt rollback to previous version (simulation) attempt_rollback() { echo "" echo "--------------------------------------------" echo "Attempting Rollback" echo "--------------------------------------------" read -p "Rollback to the previous version ($CURRENT_VERSION) ? (y/N): " ROLLBACK_CONFIRM if [[ "$ROLLBACK_CONFIRM" =~ ^[Yy]$ ]]; then echo "Initiating rollback to version $CURRENT_VERSION..." az aks upgrade --resource-group "$RESOURCE_GROUP" --name "$CLUSTER_NAME" --kubernetes-version "$CURRENT_VERSION" --yes if [ $? -eq 0 ]; then echo "Rollback executed successfully." else echo "Rollback failed. Please check the error messages above and consider manual recovery." exit 1 fi else echo "Rollback aborted. Please perform manual recovery if necessary." exit 1 fi } #--------------------------- # Main Script #--------------------------- # Prompt for input parameters read -p "Enter the Resource Group: " RESOURCE_GROUP read -p "Enter the AKS Cluster Name: " CLUSTER_NAME read -p "Enter the AKS Region (e.g., eastus): " LOCATION # Download cluster credentials dynamically echo "" echo "Downloading cluster credentials for '$CLUSTER_NAME' in resource group '$RESOURCE_GROUP'..." az aks get-credentials --resource-group "$RESOURCE_GROUP" --name "$CLUSTER_NAME" --overwrite-existing ############################################################################### # Step 1: Retrieve and Display the Current Kubernetes Version ############################################################################### echo "" echo "Fetching the current Kubernetes version for cluster '$CLUSTER_NAME' in '$RESOURCE_GROUP'..." CURRENT_VERSION=$(az aks show --resource-group "$RESOURCE_GROUP" --name "$CLUSTER_NAME" --query "kubernetesVersion" -o tsv) if [ -z "$CURRENT_VERSION" ]; then echo "ERROR: Failed to retrieve the current Kubernetes version. Please check your cluster details." exit 1 fi echo "Current Kubernetes version: $CURRENT_VERSION" echo "" ############################################################################### # Step 2: Retrieve Allowed Upgrade Paths for the Cluster ############################################################################### echo "Retrieving allowed upgrade paths for your cluster..." UPGRADES_JSON=$(az aks get-upgrades --resource-group "$RESOURCE_GROUP" --name "$CLUSTER_NAME" -o json) ALLOWED_UPGRADES=$(echo "$UPGRADES_JSON" | jq -r 'if .controlPlaneProfile.upgradeProfile.upgrades then (.controlPlaneProfile.upgradeProfile.upgrades | map(.kubernetesVersion) | join(" ")) else "" end') # Fallback: if no allowed upgrades are determined and CURRENT_VERSION starts with "1.30" if [ -z "$ALLOWED_UPGRADES" ]; then if [[ "$CURRENT_VERSION" =~ ^1\.30 ]]; then echo "WARNING: No allowed upgrade paths could be determined automatically." echo "Typically, if your cluster is running a version like 1.30.x (e.g., 1.30.10)," echo "you can only upgrade directly to a 1.31.x version." echo "For example, allowed upgrade targets might include: 1.31.6, 1.31.5, 1.31.4, 1.31.3, 1.31.2, or 1.31.1." ALLOWED_UPGRADES="1.31.6 1.31.5 1.31.4 1.31.3 1.31.2 1.31.1" ALLOWED_MAJOR_MINOR="1.31" else echo "WARNING: No allowed upgrade paths could be determined automatically. Proceed with caution." ALLOWED_MAJOR_MINOR="" fi else # Extract unique major.minor values from allowed upgrades ALLOWED_MAJOR_MINOR=$(for ver in $ALLOWED_UPGRADES; do echo "$ver" | awk -F. '{print $1"."$2}'; done | sort -u | tr '\n' ' ') fi if [ -n "$ALLOWED_MAJOR_MINOR" ]; then echo "" echo "Based on your current version ($CURRENT_VERSION), you can upgrade directly to versions with major.minor:" for mm in $ALLOWED_MAJOR_MINOR; do echo " - $mm" done echo "Only versions matching these allowed major.minor values will be marked as [ALLOWED] below." echo "For more details, please see https://aka.ms/aks-supported-k8s-ver" echo "" fi ############################################################################### # Step 3: Fetch Available Kubernetes Versions ############################################################################### echo "Fetching available Kubernetes versions in '$LOCATION'..." VERSIONS_JSON=$(az aks get-versions --location "$LOCATION" -o json) if [ $? -ne 0 ]; then echo "ERROR: Failed to fetch available versions. Please check your Azure CLI configuration." exit 1 fi # Extract list of versions and preview flag from the "values" array mapfile -t OPTIONS < <(echo "$VERSIONS_JSON" | jq -r '(.values // [])[] | "\(.version) \(.isPreview)"') if [ ${#OPTIONS[@]} -eq 0 ]; then echo "ERROR: No versions found for location '$LOCATION'." echo "This might be due to the aks-preview extension altering the output." echo "If you don't need preview features, try removing the extension with: az extension remove --name aks-preview" exit 1 fi ############################################################################### # Step 4: Build the Interactive Menu with Highlighted Allowed Options ############################################################################### declare -a VERSION_LIST declare -a LABELS for entry in "${OPTIONS[@]}"; do # Extract version and preview flag VERSION=$(echo "$entry" | awk '{print $1}') IS_PREVIEW=$(echo "$entry" | awk '{print $2}') LABEL="$VERSION" if [ "$IS_PREVIEW" == "true" ]; then LABEL="$LABEL (Preview)" else LABEL="$LABEL (Stable)" fi # Highlight if allowed (by comparing major.minor) if [ -n "$ALLOWED_MAJOR_MINOR" ]; then AVAILABLE_MM=$(echo "$VERSION" | awk -F. '{print $1"."$2}') for allowed in $ALLOWED_MAJOR_MINOR; do if [ "$AVAILABLE_MM" == "$allowed" ]; then LABEL="$LABEL [ALLOWED]" break fi done fi VERSION_LIST+=("$VERSION") LABELS+=("$LABEL") done echo "" echo "Select the Kubernetes version to upgrade to:" PS3="Enter your choice (or type 'q' to quit): " select opt in "${LABELS[@]}"; do if [[ "$REPLY" == "q" ]]; then echo "Exiting..." exit 0 fi if [ -z "$opt" ]; then echo "Invalid selection. Please try again." else TARGET_VERSION=${VERSION_LIST[$((REPLY-1))]} echo "You selected: $opt" break fi done ############################################################################### # Step 5: Validate the Selected Target Version ############################################################################### if [ -n "$ALLOWED_MAJOR_MINOR" ]; then AVAILABLE_MM=$(echo "$TARGET_VERSION" | awk -F. '{print $1"."$2}') ALLOWED_MATCH=0 for allowed in $ALLOWED_MAJOR_MINOR; do if [ "$AVAILABLE_MM" == "$allowed" ]; then ALLOWED_MATCH=1 break fi done if [ $ALLOWED_MATCH -ne 1 ]; then echo "" echo "WARNING: Upgrading from $CURRENT_VERSION to $TARGET_VERSION is not allowed based on your cluster's upgrade policy." echo "Allowed upgrades from your current version are only for versions with major.minor:" for mm in $ALLOWED_MAJOR_MINOR; do echo " - $mm" done echo "Please select one of these allowed versions." exit 1 fi fi ############################################################################### # Step 6: Pre-Upgrade Health & Compatibility Checks ############################################################################### perform_pre_upgrade_checks perform_compatibility_checks ############################################################################### # Step 7: Confirm and Execute the Upgrade ############################################################################### echo "" read -p "Proceed with upgrading '$CLUSTER_NAME' from version $CURRENT_VERSION to $TARGET_VERSION? (y/N): " CONFIRM if [[ ! "$CONFIRM" =~ ^[Yy]$ ]]; then echo "Upgrade cancelled." exit 0 fi echo "" echo "Initiating upgrade to version $TARGET_VERSION..." az aks upgrade --resource-group "$RESOURCE_GROUP" --name "$CLUSTER_NAME" --kubernetes-version "$TARGET_VERSION" --yes if [ $? -eq 0 ]; then echo "Upgrade command executed successfully." else echo "ERROR: Upgrade command failed. Please check the error messages above." exit 1 fi ############################################################################### # Step 8: Post-Upgrade Health Checks & Rollback Option # Rollback Mechanism Note: The rollback feature in this script is designed to simulate a recovery process by re-upgrading the cluster back to its previous version. # Please note that AKS does not officially support downgrades, so this rollback is not a true downgrade in the traditional sense. It is a best-effort approach that relies on having a known, working previous version and should only be used as a last resort. # Ensure that you have proper backups and recovery strategies in place before relying on this functionality. ############################################################################### if ! perform_post_upgrade_checks; then echo "" echo "One or more post-upgrade health checks have failed." attempt_rollback else echo "" echo "Post-upgrade health checks passed. Your applications appear healthy." fi echo "" echo "Upgrade complete. Please continue monitoring your cluster and applications for any issues." You can download the full script here: Custom CLI Script for AKS Upgrades Additional considerations Rollback mechanism note: The rollback feature in this script is designed to simulate a recovery process by re-upgrading the cluster to its previous version. Please note that AKS does not officially support downgrades, so this rollback is not a true downgrade in the traditional sense. It is a best-effort approach that relies on having a known, working previous version and should only be used as a last resort. Ensure you have proper backups and a comprehensive recovery strategy in place before relying on this functionality. Final thoughts Microsoft’s official AKS upgrade features are powerful and designed to simplify the process—from automated cluster and node image upgrades to orchestrated updates across multiple clusters using Fleet. For most users, these built-in capabilities offer the most reliable and supported approach. That said, if you’re a user with unique requirements, exploring a custom solution can provide granular control over every step of the process. This custom CLI script is provided as a proof-of-concept to inspire those who wish to tailor the upgrade process to their specific needs—but always remember, use it at your own risk. It is experimental, not production-ready, and is not fully supported by Microsoft.Migrating from OpenAI to Azure OpenAI
Azure OpenAI is the new destination for your AI projects that were previously on OpenAI. By migrating to Azure OpenAI, you can benefit from improved data privacy, security, and customization options. Azure OpenAI provides a secure, private, and fully controlled environment, where your data and AI models are only accessible to you, supported by Microsoft's strong infrastructure and rigorous privacy protocols.
