A year ago, teams were still asking whether Kubernetes belonged anywhere near their AI workloads. That question has mostly been answered as the community comes together to . The real challenge now is operational: how to run training and inference at scale when cost, latency, and reliability are constantly in tension.
The AKS announcements at Microsoft Build focus on that tension directly. They expand what you can control across the stack: how individual clusters are operated, how close workloads run to the hardware, how you scale across a fleet, and how AI workloads are trained and served on top.
Start with the cluster: reduce undifferentiated work
A lot of Kubernetes effort still goes into maintaining the cluster itself rather than running applications. Two updates reduce that overhead and make cluster behavior more predictable.
Managed system node pools in AKS Automatic are now generally available. These nodes run the core components that keep the cluster healthy. Previously, you had to plan capacity, handle patching, and scale them manually. Now Azure handles that lifecycle. In practice, this means system components no longer compete with your workloads for resources, which is especially important for GPU-backed nodes. That separation gives you more consistent performance and fewer surprises when workloads scale.
Azure Container Linux, also generally available, standardizes the host OS layer. It is a minimal, container-optimized Linux distribution maintained by Microsoft. The smaller package footprint reduces patching overhead and limits drift across clusters. For teams operating multiple environments, this creates a consistent baseline that is easier to maintain and secure over time, which becomes more important as fleets grow.
Tune for performance: remove infrastructure bottlenecks
Some workloads benefit from abstraction, while others pay for it (especially when you are pushing hardware limits).
AKS on bare metal, now in public preview, is designed for those cases. It lets you run AKS on dedicated machines without a hypervisor, giving you direct access to NVLink, RDMA, and high-performance networking. You still use the same AKS control plane and APIs, but without the additional layer between your workloads and the hardware.
This matters for large training jobs, latency-sensitive inference, and high-throughput data pipelines, where small inefficiencies translate directly into higher cost or longer runtimes. Removing that overhead improves both performance and utilization as you scale.
Scale across the fleet: operate environments as one system
Most teams are not managing a single cluster – they are managing many, often across regions, clouds, and on-prem environments, and those differences tend to show up as operational inconsistency.
Azure Kubernetes Fleet Manager for Arc-enabled clusters is now generally available and extends fleet-level control beyond Azure. You can apply updates, enforce policies, and place workloads across clusters from a single control plane, which reduces the need to manage each environment independently.
For AI workloads, this shows up in a few practical ways. You can roll out changes progressively with health checks between stages, place workloads based on GPU availability and SKU, and apply RBAC consistently across environments. That consistency makes it easier to reason about reliability and capacity as systems scale.
The AI layer: training and inference as first-class primitives
With the cluster and fleet in place, the next challenge is coordinating distributed AI workloads efficiently.
Anyscale on Azure, now in public preview, brings managed Ray to AKS. Kubernetes continues to handle scheduling and cluster lifecycle, while Ray coordinates distributed execution within a workload. That includes managing CPUs and GPUs together, handling heterogeneous and fractional GPU allocation, and orchestrating jobs that scale dynamically based on demand.
The service runs inside your Azure subscription, integrates with Entra ID, and is billed through your existing agreement. It gives you more control over how compute is allocated within a job, which directly impacts both cost efficiency and time to completion.
Model serving: from YAML to production endpoints
Model serving has been one of the more complex parts of running AI on Kubernetes. Open source tooling is starting to simplify that path without replacing the underlying platform.
AI Runway, introduced earlier this year at KubeCon Europe, provides a Kubernetes-native way to deploy and operate models. Instead of starting with configuration, you start by selecting a model, validating that it fits your available GPU memory, reviewing cost estimates, and deploying it. That action creates a ModelDeployment custom resource, which drives the rest of the workflow.
KAITO, the Kubernetes AI Toolchain Operator, handles much of the underlying orchestration. It estimates resource requirements, provisions nodes through tools like Karpenter, and launches optimized runtimes such as vLLM. From there, standard Kubernetes components take over. KEDA manages autoscaling based on workload metrics, Gateway API handles routing, and rollout strategies follow familiar Kubernetes patterns.
AI Runway sits as a platform layer, with KAITO as one of several providers alongside options like NVIDIA Dynamo and KubeRay. It supports multiple serving engines, including vLLM, SGLang, TensorRT-LLM, and llama.cpp. These tools build on Kubernetes primitives rather than hiding them, so you can move faster without giving up visibility or control.
What this looks like in production
These capabilities show up as consistent patterns across teams, even when the use cases differ.
Wayve is building an end-to-end deep learning AI Driver that can be fine-tuned to drive almost any car, in any city, primarily from cameras. It trains and validates that model on Azure, using AKS, Ray and now Anyscale on Azure to connect thousands of GPUs into a flexible supercomputer. The approach generalizes: Wayve took a new Nissan vehicle in Tokyo, a city it had never driven, and had it driving autonomously in four months. Wayve-equipped vehicles now operate in the UK, the United States, Germany, and Japan.
Royal Bank of Canada built a self-service AI platform on AKS where development teams provision GPU resources and deploy models through a CI/CD workflow they run themselves. KAITO handles production model serving, with model images hosted in the bank's private container registry. The compliance perimeter wraps the entire path: private endpoints, Entra ID, Key Vault, and a private ACR keep models and data inside the bank's Azure boundary. Developers get self-service speed; the business gets the security and audit trail it requires.
SimCorp unified its SimCorp One investment management platform on Azure and standardized on AKS to run workloads consistently across regions, without configuration drift and without turning Kubernetes into a product it has to operate. With that in place, it embedded governed, auditable AI directly into investment workflows.
The practitioner's takeaway
The common thread across these updates is not that they remove the cost-latency-reliability tradeoffs, but that they make them easier to manage explicitly.
At the cluster layer, you trade control for simplicity where it makes sense, such as offloading system node management. At the infrastructure layer, you can choose between flexibility and raw performance, including when to run directly on hardware. At the workload layer, you decide how tightly to control scheduling, scaling, and serving behavior based on the needs of your models.
The practical approach is to start with the constraint that matters most for your workload. If cost is the issue, focus on utilization and right-sizing. If latency is the constraint, look at placement and hardware access. If reliability is the concern, prioritize rollout controls and fleet consistency. Azure Kubernetes Service expands your options in each of these areas so you can make those tradeoffs deliberately rather than working around them.
For the deeper version of this, join The honest practitioner's take on agentic AI on Kubernetes (BRK222) at Microsoft Build 2026, live or online.
Microsoft