Startups at Microsoft

9 MIN READ

AI on Azure: when you own the infrastructure

Microsoft

Oct 30, 2026

AI on Azure series, Part 5 of 5

Many teams can use a hosted model API or managed ML service and leave much of the infrastructure work to the provider. Others train large models, serve open-weight models at high volume, or need to choose the GPU type, scheduler, network, and storage themselves.

Those workloads can run on Azure infrastructure. Your team gets more control and takes on more operational work. A VM family, a Kubernetes runtime and a scheduler solve different problems, so the options below are grouped by their role rather than ranked.

Product details were checked against Microsoft Learn on October 29, 2026. Preview labels apply to individual features, not necessarily to the whole product. Check each feature's support terms and production guidance before relying on it.

1. What you take on

Using VMs or Kubernetes gives your team more control over the environment, but your team must manage drivers, images, scheduling, failures, capacity, and cost. Choose that approach when you need control over a specific GPU generation, topology, or serving stack.

2. GPU virtual machines

Azure groups GPU VMs by purpose. The names look similar, but the capabilities differ.

Family	Typical role
ND	Large-scale training and high-throughput inference. Includes ND H100 v5, ND H200 v5, ND GB200 v6 and ND GB300 v6, plus AMD MI300X and earlier A100 sizes.
NC	Applied AI, batch inference, and ML development across a range of GPU configurations. Includes NCads H100 v5, NC A100 v4 and NCasT4 v3.
NV	Visualization and virtual desktop workloads. Not the usual choice for AI training.

The series pages show why the exact SKU matters:

ND H100 v5 has 8 H100 GPUs with a dedicated 400 Gbps InfiniBand connection per GPU. ND H200 v5 has 8 H200 GPUs with 141 GB each.
ND GB200 v6 and ND GB300 v6 have 4 Blackwell GPUs per VM and are deployed in NVL72 rack configurations. The GB300 page positions the series for high-throughput inference.
ND GB200 and GB300 do not support Live Migration, Memory Preserving Updates, Service Healing or Nested Virtualization. Plan your fault handling around that.
Don't infer multi-node capability from the family name. Check the exact VM size for its GPU count, interconnect and scale-out support.
NVv3 and NVv4 retired on September 30, 2026. Check the current NV family page for supported series and regional availability.

Check the individual series pages for current hardware and feature support, and confirm availability in your target region.

Sources: GPU VM sizes, ND family, NC family, NV family, ND H100 v5, ND GB300 v6, ND GB200 v6, ND H200 v5.

3. Kubernetes with GPUs: AKS

Evaluate AKS when you need containers, autoscaling, and a shared platform for multiple teams.

GPU node pools. On the self-managed GPU path, AKS installs the NVIDIA driver by default; you manage the device plugin, metrics exporter, and GPU health tooling. If you use the NVIDIA GPU Operator, follow AKS guidance to disable AKS driver installation so the operator can manage the driver stack, including its compatibility and maintenance. AKS also offers a full managed GPU stack, including the driver, device plugin, DCGM metrics exporter, and health signals, in Preview. This Preview is excluded from AKS SLAs and isn't intended for production. It works with node auto-provisioning, including AKS Automatic. The managed and self-managed paths have different operational boundaries; choose and document one.

You cannot add a GPU VM size to an existing general-purpose node pool. Create a new pool.
AKS supports MIG on selected A100, H100, and H200 sizes. The standard multi-instance node-pool guide has you install the NVIDIA device plugin; the separate fully managed GPU-node/MIG stack is in Preview. The GPU partitioning overview describes managed MIG as production-capable, so verify which path it means before making a production claim. Time-slicing and MPS are user-managed through the NVIDIA GPU Operator, do not provide hardware isolation, and aren't recommended for workloads that require strict SLAs.
AKS stopped supporting Azure Linux 2.0 and providing security updates on November 30, 2025. Its node images are scheduled for removal on October 31, 2026, after which affected pools cannot scale. Use a supported OS image.

KAITO. The KAITO add-on deploys inference workspaces from a Kubernetes resource. It integrates with vLLM, exposes an OpenAI-compatible API, and uses Karpenter to provision GPU nodes from the model's workspace specification. It requires NVIDIA Ampere or newer, a public Azure region, and GPU VM quota in that region. GPU node pools provisioned for a workspace are not deleted automatically when you remove it. AKS stop and start operations are not fully supported while KAITO workspaces are active. The current Learn guide does not state a GA or Preview label, so confirm lifecycle status before presenting KAITO as production-ready.

vLLM and Triton. Both are serving runtimes you run in containers. They are not Azure services. The AKS GPU reference architecture mentions both.

Sources: Use GPUs on AKS, multi-instance GPU node pools, GPU partitioning on AKS, Managed GPU node pools (Preview), AKS node image retirements, KAITO add-on, AKS GPU reference architecture, Managed KubeRay (Preview), Ray and Kueue on AKS.

Ray on AKS: three operating models

Ray is an open-source runtime for distributed Python and AI workloads, including training, batch inference, and online serving. KubeRay manages Ray resources on Kubernetes; Kueue adds workload admission and queueing. These components have different jobs, and running them on AKS does not automatically make the whole stack a Microsoft-managed service.

Path	What it provides	What remains with your team
Open-source Ray with KubeRay and Kueue	Microsoft's Terraform-based infrastructure guide provisions AKS and installs the operators through Helm.	Operate the deployed operators and Ray workloads, manage runtime images and upgrades, and arrange open-source support. The guide excludes software you deploy from AKS SLAs and Azure support.
Managed KubeRay extension, Preview	AKS installs and manages the KubeRay operator and Ray custom resource definitions.	Manage Ray workloads and runtime images. The extension is excluded from AKS SLAs and is not intended for production.
Anyscale on Azure	An Anyscale-operated control plane and operator manage Ray cluster lifecycles, with a shared-responsibility model for the clusters in your subscription.	Own the application and Azure data-plane resources; review the AKS, storage, networking, and access responsibilities rather than assuming the entire stack is outsourced.

Anyscale on Azure is a separate Azure Native Integration, not another name for the managed KubeRay extension. Its architecture and responsibility matrix describe the boundary between Anyscale's control plane and the Ray workloads on your AKS cluster. Part 4 covers it as a managed platform option. The extension's Preview status does not establish the lifecycle or support terms of the Anyscale offer.

The January 2025 AKS announcement introduced open-source Ray guidance and commercial RayTurbo support on AKS. Use it as historical context, not as the current support matrix for Anyscale on Azure. Check the current offer's regions, feature limitations, pricing, and support terms before selecting a deployment.

4. Training clusters: Batch, CycleCloud and Slurm

Batch and CycleCloud provide different operating models:

Azure Batch is a scheduler as a service. There is no extra charge beyond the underlying VMs, storage and networking. It supports independent tasks and tightly coupled MPI jobs, including multi-node training.
Azure CycleCloud is not a scheduler. It provisions and autoscales HPC infrastructure, and you operate the scheduler (Slurm, PBS Pro, LSF, Grid Engine or HTCondor).
CycleCloud Workspace for Slurm is a Marketplace solution that deploys CycleCloud, Slurm, networking, storage and access together.

Using Slurm alone is not a reason to choose CycleCloud, and avoiding VM management alone is not a reason to choose Batch. Document why the option fits your workload and calculate peak quota before production.

Sources: Azure Batch technical overview, Azure CycleCloud overview.

5. Serverless GPUs: Azure Container Apps

Container Apps serverless GPUs scale to zero and use per-second billing. The model and serving setup must fit the selected GPU and service limits:

Available only in workload profiles environments, not Consumption-only environments, with A100 and T4 GPUs.
A100 availability is limited to a set of regions, so check yours.
A replica cannot use fractional or multiple GPUs, and only the first container in the app can access its GPU.
Image startup time can dominate cold starts. Artifact streaming can help, but it requires Azure Container Registry Premium.

Deploying supported Foundry models to Container Apps is Preview. Check that the model fits one GPU, then test whether scaling replicas meets the workload's demand.

Source: Serverless GPUs in Container Apps.

6. Storage for training

Training jobs can push a lot of data through storage, so keep active job data separate from files you need only after a run.

Option	Use it for
Azure Managed Lustre	Hot, parallel file system for training data and checkpoints, with Blob import and export. Microsoft's guidance picks it for the lowest transfer times.
Azure NetApp Files	Enterprise features, tuning control, or an existing NetApp investment.
Local NVMe	Fast temporary scratch data on VM sizes that include local NVMe. The data is ephemeral, so plan how to checkpoint or copy it elsewhere.
Azure Blob Storage with NFS 3.0	Read-heavy, sequential, high-throughput access. Needs a hierarchical namespace. Not for transactional workloads or in-place edits.

With Blob NFS 3.0, traffic must come from a virtual network. It doesn't support account-key, Microsoft Entra ID or ACL authorization, and it doesn't support rename or link operations.

Save checkpoints regularly and move inactive data to Blob Storage. Blob versioning can help preserve data versions on supported account types, but it isn't supported on hierarchical-namespace accounts, which NFS 3.0 requires. Check the Blob versioning support matrix before combining these features.

Sources: Storage for AI infrastructure, Azure Managed Lustre, Blob NFS 3.0.

7. Quota and capacity

Plan both quota and physical capacity before settling on a GPU architecture.

GPU quota is enforced per subscription, per region, at two levels: total regional vCPUs and the VM-family vCPU quota. Increases for adjustable quotas go through the Quotas page, and some require a support request.
A quota approval is not a capacity guarantee. On-demand capacity reservations can secure capacity with an SLA for supported configurations. They need quota and are billed at the VM rate even when unused.
Reservation support varies by VM series. Current docs list NC A100 v4 and several NV series as supported, but exclude the ND series and NCads H100 v5. Check the current list for the exact size you need, and talk to your Microsoft account team early for large ND capacity.
Reserved Instances discount the price but do not guarantee capacity.

Sources: VM-family quota increases, Capacity reservation.

8. Choosing

If you need	Start by evaluating
Multi-node training with a scheduler you already run	CycleCloud Workspace for Slurm
Training or batch jobs without managing a scheduler	Azure Batch
A shared container platform for training and serving	AKS with GPU node pools
Distributed Python or AI workloads with a team-operated Ray stack	Ray with KubeRay on AKS, with Kueue when workload admission and queueing are needed
Ray workloads with a managed platform and commercial support	Anyscale on Azure; see Part 4 for its shared responsibilities
Fast, managed open-weight model serving on AKS	KAITO
Inference with variable demand and a model that fits one supported GPU	Container Apps serverless GPU

Confirm the choice against your workload's GPU generation, interconnect needs, region, quota and data location.

Before you commit

Which layer do you want your team to operate, and why?
Which exact VM size and region, and do you have the family quota?
Do you need a capacity guarantee, and is that VM series eligible?
Where does training data live, and how do checkpoints get saved?
Who owns drivers, images and open-source runtime upgrades?

AI on Azure series

Part 1: AI on Azure 101
Part 2: Models as a service
Part 3: Platforms for apps and agents
Part 4: Managed ML
Part 5: When you own the infrastructure (this article)

Prices are public rates. Your agreement may differ, so use the Azure pricing calculator and confirm with your account team.

References

Updated Oct 30, 2026

Version 2.0

rmmartins