Inference Options on Azure: An Architect’s View

A snapshot, as of October 2026. Azure's inference options keep evolving, so the number of options may change, and there may be more in the future. This post covers what's available today.

This guide compares six options for serving a model on Azure, from fully managed to fully self-operated, with a short runnable demo for the first five. Every demo uses the same Python client, and only the endpoint changes.

What can be served is the first filter. Closed, proprietary models such as GPT and Claude are available only as a service (Option 1); their weights are never available to run yourself. Options 2 to 6 serve open-weight (OSS) models or weights you own. If you need a frontier proprietary model, the choice is made for you and it's a fully managed PaaS service. If you need an open or custom model, choose among options 2 to 6 by how much you want to operate and how you want to pay. Some open-weight models, such as Llama and DeepSeek, are also offered through Option 1 as pay-per-token Foundry Models.

Option 6 is included for comparison but doesn't yet have a runnable demo in this guide.

 

 

Deployment options in Foundry

Microsoft Learn's deployment overview defines two Foundry deployment options:

• Serverless API: the preferred, most capable path, for Foundry Models sold by Azure and selected partner models. It includes the standard, provisioned throughput, batch, and developer deployment types.
• Managed compute (preview): open-source, partner, and custom models on dedicated GPU capacity that Foundry manages for you.

Foundry also offers instant access (preview), which calls supported models by name without creating any deployment.

Foundry also integrates with Fireworks on Foundry, a first-party Microsoft consumption service that runs open-source catalog models and imported custom weights on Fireworks' proprietary inference engine and GPU infrastructure. It supports pay-per-token Standard deployments and provisioned throughput, uses an Azure meter, and requires a subscription administrator to opt in.

Outside Foundry, Azure gives you two more places to run a model: Azure Container Apps serverless GPU, where you bring a container and Azure provides on-demand GPUs, and AKS, where you run the cluster yourself. The Container Apps documentation describes serverless GPU as "a middle layer option between the Foundry model catalog's serverless APIs and hosting models on managed compute".

Where these options fit in the Azure picture

The managed compute overview has a table of Foundry deployment types. The table below extends it with the non-Foundry options this guide demos (rows marked ➕).

The Foundry paths use Foundry project governance and Azure billing. Fireworks on Foundry remains a Microsoft service, but Fireworks processes requests on its GPU infrastructure as Microsoft's subprocessor. The three ➕ rows each add a separate endpoint and operating model. The /v1 OpenAI-compatible API gives all six the same application-facing request shape.

Prerequisites

• Azure CLI 2.60 or later, logged in (az login), and kubectl for options 4 and 5.
• Python 3 with pip.
• For options 2, 4, and 5: GPU quota for the SKU you use. Option 2 uses Container Apps managed-environment GPU quota, which is separate from VM quota.
• For option 3: managed compute quota for the accelerator family. It is granted through the Foundry quota process and is separate from Azure VM quota.

Shared variables:

export LOC=southcentralus
export RG=rg-inferencing-demo
export GPU_VM=Standard_NC4as_T4_v3          # options 4 and 5
az group create -n $RG -l $LOC

For options 4 and 5, confirm that your chosen GPU VM SKU is available in the target region, you have sufficient VM-family and regional vCPU quota, and your organization's Azure Policy allows that SKU. These are three separate checks. Options 1, 2, and 3 don't use AKS GPU VM quota.

The test client

Every demo is called with the same local chat.py. It uses the plain openai SDK and supports either an API key or a Microsoft Entra ID token (set OPENAI_API_KEY=entra).

The client and its dependencies are included inline below; no repository checkout is needed.

Run the following block from your working directory to write the inline code to chat.py. It replaces the file with the same code included in this guide.

cat <<'EOF' > chat.py
import os
from openai import OpenAI

base = os.environ["OPENAI_BASE_URL"]
key = os.environ.get("OPENAI_API_KEY", "")

if key in ("", "entra", "aad"):
    from azure.identity import DefaultAzureCredential, get_bearer_token_provider
    token = get_bearer_token_provider(DefaultAzureCredential(), "https://ai.azure.com/.default")
    client = OpenAI(base_url=base, api_key=token)
else:
    client = OpenAI()  # key auth: reads OPENAI_BASE_URL + OPENAI_API_KEY

resp = client.chat.completions.create(
    model=os.environ["OPENAI_MODEL"],
    messages=[{"role": "user", "content": "What is Kubernetes, in one sentence?"}],
    max_tokens=64, temperature=0,
)
print(f"[endpoint] {base}")
print(resp.choices[0].message.content)
EOF

Install the client dependencies from the same working directory:

cat <<'EOF' > requirements.txt
# Dependencies for the local inference client and managed compute deployment.
# openai >= 1.106 is required to pass an Entra token provider as api_key.
openai>=1.106.0
azure-identity>=1.19.0
# Preview management SDK version used by the managed compute how-to.
azure-mgmt-cognitiveservices==15.0.0b2
EOF
python3 -m pip install -r requirements.txt

 

Option 1: Foundry serverless API

The most managed option. You never see a GPU, an image, or a runtime; you pick a model and a deployment type, and you pay per token.

📘 Terminology

• Foundry resource: an AIServices Cognitive Services account that hosts model deployments.
• Model deployment: a named, callable instance of a catalog model behind an endpoint.
• GlobalStandard: pay-per-token capacity; nothing to provision.

Step 1.1: Create the resource and deploy a model

Choose a globally unique name for your Foundry resource and replace <your-foundry-resource-name> below before running the commands. Save your chosen name and use the same value on every rerun and in each new terminal session; do not generate a new name for each test.

export AOAI="<your-foundry-resource-name>"

az cognitiveservices account create \
  -n $AOAI -g $RG -l $LOC \
  --kind AIServices --sku S0 \
  --custom-domain $AOAI --yes

# Model versions retire, so list available OpenAI model versions whose names contain 'mini':
az cognitiveservices account list-models -g $RG -n $AOAI \
  --query "[?format=='OpenAI' && contains(name,'mini')].{name:name, version:version}" -o table

az cognitiveservices account deployment create \
  -g $RG -n $AOAI \
  --deployment-name gpt-4.1-mini \
  --model-name gpt-4.1-mini --model-version 2025-04-14 --model-format OpenAI \
  --sku-name GlobalStandard --sku-capacity 10

Already created it? Set AOAI to that resource's exact name and verify it with az cognitiveservices account show -g "$RG" -n "$AOAI" -o table, then skip the account creation command. If the model deployment already exists too, continue with Step 1.2.

Step 1.2: Grant data-plane access and call it

Use Entra ID for authenticating vs using key based:

export AOAI_ID=$(az cognitiveservices account show -g $RG -n $AOAI --query id -o tsv)
az role assignment create \
  --assignee $(az ad signed-in-user show --query id -o tsv) \
  --role "Cognitive Services OpenAI User" \
  --scope "$AOAI_ID"

export OPENAI_BASE_URL="https://$AOAI.openai.azure.com/openai/v1/"
export OPENAI_API_KEY=entra
export OPENAI_MODEL=gpt-4.1-mini

python3 chat.py

 

What you get: an answer in under a minute, with no GPU quota, no YAML, and no runtime to patch.

What you give up: you can only use catalog models, and you can't change the GPU, the engine, or its flags.

Option 2: Azure Container Apps serverless GPU

You bring the container; Azure brings the GPU, and only while it's needed. There's no cluster, node pool, driver, or device plugin. You still own the serving runtime, so this demo runs the same vLLM image and flags as Option 5, on the same NVIDIA T4 class of GPU.

What makes it different from every other option here:

• Scale to zero. With minReplicas: 0, an idle app releases its GPU and you stop paying for it.
• Per-second billing. You pay for GPU, vCPU, and memory seconds only while a replica runs.
• Any bring your own model. You run whatever container you build, including fine-tunes and custom engines.

📘 Terminology

• Workload profiles environment: a Container Apps environment that can host several compute profiles. Serverless GPUs require one; a Consumption-only environment doesn't support them.
• Consumption GPU profile: Consumption-GPU-NC8as-T4 (1 × T4, 8 vCPU, 56 GiB) or Consumption-GPU-NC24-A100 (1 × A100, 24 vCPU, 220 GiB). Check regional availability first.
• Revision: an immutable snapshot of the app's template. Scale settings are part of it.

Step 2.1: Check the GPU profile, then create the environment

export ACA_ENV=aca-demo-gpu-env
export ACA_APP=phi-vllm-aca

# Which serverless GPU profiles does this region offer?
az containerapp env workload-profile list-supported -l $LOC \
  --query "[?contains(name,'GPU')].{name:name, cores:properties.cores, memGiB:properties.memoryGiB, gpus:properties.gpus}" \
  -o table

az containerapp env create -g $RG -n $ACA_ENV -l $LOC --enable-workload-profiles

az containerapp env workload-profile add -g $RG -n $ACA_ENV \
  --workload-profile-name gpu-t4 \
  --workload-profile-type Consumption-GPU-NC8as-T4

 

 

Step 2.2: Deploy vLLM on the serverless GPU

The template is the same vllm/vllm-openai container and flags as Option 5, with four Container Apps-specific choices:

• API key. Container Apps ingress is a public HTTPS endpoint, unlike the port-forwarded AKS services, so vLLM reads an API key from a secret (VLLM_API_KEY). /health stays open for probes; /v1/* requires the key.
• Startup probe. vLLM opens port 8000 only after it downloads and loads the weights. This probe allows up to 11 minutes (60 s delay + 10 × 60 s).
• minReplicas: 0 enables scale to zero.
• cooldownPeriod: 900 keeps an idle replica for 15 minutes before scaling in. The default of 300 s is shorter than a vLLM cold start.

export ACA_ENV_ID=$(az containerapp env show -g $RG -n $ACA_ENV --query id -o tsv)
export VLLM_KEY=$(openssl rand -hex 16)

cat <<EOF | az containerapp create -g $RG -n $ACA_APP --yaml /dev/stdin
location: $LOC
properties:
  environmentId: $ACA_ENV_ID
  workloadProfileName: gpu-t4
  configuration:
    secrets:
      - name: vllm-api-key
        value: $VLLM_KEY
  template:
    containers:
      - name: vllm
        image: docker.io/vllm/vllm-openai:v0.31.0
        args:
          - "--model=microsoft/Phi-3.5-mini-instruct"
          - "--served-model-name=phi-3.5-mini"
          - "--dtype=half"
          - "--max-model-len=4096"
          - "--gpu-memory-utilization=0.9"
          - "--trust-remote-code"
        env:
          - name: VLLM_API_KEY
            secretRef: vllm-api-key
        resources:
          cpu: 8
          memory: 56Gi
        probes:
          - type: Startup
            httpGet: { path: /health, port: 8000 }
            initialDelaySeconds: 60
            periodSeconds: 60
            failureThreshold: 10
          - type: Readiness
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 10
    scale:
      minReplicas: 0
      maxReplicas: 1
      cooldownPeriod: 900
EOF

az containerapp ingress enable -g $RG -n $ACA_APP --type external --target-port 8000
export ACA_FQDN=$(az containerapp show -g $RG -n $ACA_APP --query properties.configuration.ingress.fqdn -o tsv)
echo "$ACA_FQDN"

 

Step 2.3: Wait for the cold start, then call it

The first replica pulls about 9 GB of image and about 7.6 GB of weights. Poll /health until it returns 200. Each request also tells the HTTP scaler there's demand:

until [ "$(curl -s -o /dev/null -w '%{http_code}' --max-time 60 https://$ACA_FQDN/health)" = "200" ]; do
  echo "$(date +%T) waiting for vLLM..."; sleep 15
done

export OPENAI_BASE_URL="https://$ACA_FQDN/v1"
export OPENAI_API_KEY=$VLLM_KEY
export OPENAI_MODEL=phi-3.5-mini

python3 chat.py

Read engine logs with az containerapp logs show -g $RG -n $ACA_APP --tail 100 --format text, and platform events (probes, scaling) with --type system.

What it cost you (vs Option 5): one environment, one GPU profile, and one app definition. No cluster, node pool, device plugin, Service, or port-forward, and the GPU goes away when idle.

What you still own: the image, the engine version and flags, the model artifact, and endpoint security. Compared with Option 3, you trade Microsoft-managed runtimes for any model and any engine, plus scale to zero.

Option 3: Foundry managed compute (preview)

Preview: no SLA, not recommended for production. Built-in content filtering isn't part of the managed compute data path yet; call Azure AI Content Safety directly if you need it.

Managed compute is a managed GPU platform for open models. You pick a model from the Foundry catalog's Hugging Face collection; Foundry provisions dedicated accelerators, runs a Microsoft-built serving runtime, and serves the model on the same Foundry endpoint as Option 1. There's no VM, cluster, container image, or runtime for you to own.

📘 Terminology

• Model instance: the unit of deployment. You don't size VMs; you set the number of instances (the capacity on the deployment SKU), and each instance uses the GPUs its template specifies.
• Deployment template: a versioned asset that pins the runtime (for example, vLLM), the accelerator family and count per instance, the context length, quantization, and runtime tuning such as tool-call and reasoning parsers, health probes, and concurrency. Choosing a template is the only knob for how a model runs.
• Accelerator family: A100_80GB, H100_80GB, or MI300_192GB in the docs. This lab's quota API also listed H200_141GB.
• GlobalManagedCompute: the deployment SKU. Managed compute currently supports Global deployment only.

Step 3.1: Select the account and check quota

Use the Python management SDK from the managed compute how-to, with the Qwen3-32B model and 40K, single-A100 configuration described in the announcement blog. The blog's SDK example uses H100. This demo selects A100 using the live catalog's allowed qwen--qwen3-32b--nvidia template, which supports either accelerator family.

You need an existing Foundry account (AIServices) and project, Cognitive Services Contributor (or the documented equivalent) on the account to deploy/delete, and Foundry User to invoke. You also need at least one free A100 in your managed compute quota; Azure VM quota doesn't count. Use your own account details below and keep the deployment name stable on reruns.

export MC_RG="rg-inferencing-demo"
export MC_ACCOUNT="<your-foundry-resource-name>"
export MC_DEPLOYMENT=qwen3-32b-demo
export SUB=$(az account show --query id -o tsv)
export MC_LOC=$(az cognitiveservices account show -g "$MC_RG" -n "$MC_ACCOUNT" --query location -o tsv)
az rest --method get \
  --url "https://management.azure.com/subscriptions/$SUB/providers/Microsoft.CognitiveServices/locations/$MC_LOC/managedComputeUsages?api-version=2026-09-01" \
  --query "value[].{family:name.value, used:currentValue, limit:limit}" -o table

Stop and resolve any errors before continuing. For a new deployment, the A100 row must have limit - used >= 1. If not, request managed compute quota in Foundry rather than switching to a more expensive accelerator.

If the account has no project yet, create one before opening its deployment in Foundry. For this lab, use the stable project name inference-demo:

export MC_PROJECT=inference-demo
az cognitiveservices account project create \
  --name "$MC_ACCOUNT" --resource-group "$MC_RG" \
  --project-name "$MC_PROJECT" --location "$MC_LOC"
az cognitiveservices account project show \
  --name "$MC_ACCOUNT" --resource-group "$MC_RG" \
  --project-name "$MC_PROJECT" --query properties.provisioningState -o tsv

Expect Succeeded. If you already have a project, use it instead. Project creation does not redeploy or scale the model. See Create a project for account identity and project-management prerequisites.

Step 3.2: Deploy with the Python SDK

Run from your working directory. The local requirements.txt pins azure-mgmt-cognitiveservices==15.0.0b2, the preview version specified by the how-to. The block writes the complete script to deploy-managed-compute.py:

python3 -m pip install -r requirements.txt

cat <<'EOF' > deploy-managed-compute.py
import os
from azure.identity import DefaultAzureCredential
from azure.mgmt.cognitiveservices import CognitiveServicesManagementClient

client = CognitiveServicesManagementClient(
    DefaultAzureCredential(), os.environ["SUB"]
)

print("Deploying one Qwen3-32B instance on one A100; typically 10-15 minutes.", flush=True)
deployment = client.managed_compute_deployments.begin_create_or_update(
    resource_group_name=os.environ["MC_RG"],
    account_name=os.environ["MC_ACCOUNT"],
    deployment_name=os.environ["MC_DEPLOYMENT"],
    resource={
        "sku": {"name": "GlobalManagedCompute", "capacity": 1},
        "properties": {
            "model": "azureml://registries/azure-huggingface/models/qwen--qwen3-32b/versions/1",
            "deploymentTemplate": "azureml://registries/azure-huggingface/deploymenttemplates/qwen--qwen3-32b--nvidia/labels/latest",
            "acceleratorType": "A100_80GB",
        },
    },
).result()

print(f"State: {deployment.properties.provisioning_state}")
print(f"ID: {deployment.id}")
if deployment.properties.provisioning_state != "Succeeded":
    raise RuntimeError("Deployment is not ready. Inspect its status before sending requests.")
EOF

python3 deploy-managed-compute.py

 

Step 3.3: Call it with the same chat.py

Managed compute deployments use the unified Foundry endpoint. If the runtime is OpenAI-compatible, which vLLM is, the deployment is reachable on /openai/v1/, and the model field is the deployment name. Calling with Entra ID requires the Foundry User role on the account:

az role assignment create \
  --assignee $(az ad signed-in-user show --query id -o tsv) \
  --role "Foundry User" \
  --scope $(az cognitiveservices account show -g $MC_RG -n $MC_ACCOUNT --query id -o tsv)

export OPENAI_BASE_URL="https://$MC_ACCOUNT.services.ai.azure.com/openai/v1/"
export OPENAI_API_KEY=entra
export OPENAI_MODEL="$MC_DEPLOYMENT"   # deployment name, not the catalog model ID

python3 chat.py

Every managed compute deployment is also reachable at https://<account>.services.ai.azure.com/managed-deployments/<deployment-name>/, which is the route for models with a non-OpenAI API.

Option 3 is not fully tested.   Instructions have been documented for comparison purposes only. 

 

Why managed compute makes many runtimes easy

Managed compute isn't just "vLLM as a service". It runs a portfolio of runtimes, each chosen per model architecture. Following table briefly explains each, research yourself if you want to understand more about them:

On AKS, each runtime is its own project. Option 5 shows the work for one runtime and one model: a GPU node pool, a device plugin, a Deployment with engine flags, probes, a Service, and a way in. Add an embeddings model and you start again with a different image, API, health endpoint, and memory profile. Add a structured-output agent model and you learn SGLang. Add a speech or vision model and you're writing Python serving code. Each one needs its own image pipeline, CVE patching, version pinning, and testing on each GPU type you run.

In managed compute, the runtime is just a field in the deployment template. Microsoft's curation pipeline does that work once per model × runtime × accelerator, before the model appears in the catalog. Runtime upgrades and CVE patches are applied to live deployments automatically.

Effort comparison: vLLM on AKS (Option 5) vs managed compute (Option 3)

Option 4: AKS with KAITO

Same result as Option 5, an open-weight model on your AKS GPUs, but the stack collapses into one Workspace CRD. KAITO (the AKS AI toolchain operator add-on) provisions the GPU node, installs the device plugin, tunes vLLM, and creates the Deployment and Service.

Step 4.1: Create a cluster with the AI toolchain operator add-on

export CLUSTER_K=aks-demo-kaito

az aks create \
  -g $RG -n $CLUSTER_K --location $LOC \
  --node-count 1 --node-vm-size Standard_D4s_v6 \
  --enable-ai-toolchain-operator --enable-oidc-issuer \
  --generate-ssh-keys

az aks get-credentials -g $RG -n $CLUSTER_K --overwrite-existing
kubectl config use-context $CLUSTER_K

 

Step 4.2: Apply one Workspace

cat <<EOF | kubectl apply -f -
apiVersion: kaito.sh/v1beta1
kind: Workspace
metadata:
  name: workspace-phi
resource:
  instanceType: "$GPU_VM"
  labelSelector:
    matchLabels: { apps: phi }
inference:
  preset:
    name: phi-3.5-mini-instruct
EOF

kubectl get workspace workspace-phi -w

 

Step 4.3: Call it

from a different terminal, setup port-forwarding with this command  kubectl port-forward svc/workspace-phi 8000:80  

export OPENAI_BASE_URL="http://localhost:8000/v1"
export OPENAI_API_KEY=not-needed
export OPENAI_MODEL=phi-3.5-mini-instruct

python3 chat.py

What it cost you (vs Option 5): one CRD instead of a node pool, device plugin, Deployment, Service, and probes.  Literally, a 5 to 7 lines of coding (Workspace definition). 

What you kept: your cluster, GPUs, network, and identity, plus the option to drop to raw YAML.

 

Check this Github repo for more information: kaito-project/kaito: Kubernetes AI Toolchain Operator

Option 5: AKS with vLLM, hand-built

The full-control end: you operate every layer, from the GPU node pool and device plugin to the model-serving runtime.  The point of this option isn't only flexibility and control. It is to see, and own, the building block that higher-level services are made of, and to feel what that ownership costs in effort.

 

Why learn this building block

Most models will run on a managed service. Building the layers once, by hand, still helps you:

• Understand how higher-level abstractions are built, so a managed endpoint stops being a black box.
• Relate a feature to how it's built. "Caching", "autoscaling", and "context length" each map to something concrete, such as a KV cache in GPU memory, a replica count, or a memory setting.
• Ask your peers the right questions, because you know which layer a claim is about.
• Have an informed conversation without staying abstract.

It's like talking about virtual machines without understanding the hypervisor, how CPU and memory are partitioned, how disks are shared, or how the software-defined network is set up. You can use VMs without it, but you can't judge why one is slow or evaluate a claim about it.

vLLM serving on a GPU shows three things:

• How one LLM call is served. The prompt is processed in a prefill step, then the answer is generated one token at a time. Weights and a KV cache share GPU memory.
• How caching works in model serving. A replica that already computed a prompt's prefix can skip that work. In the two-replica test, the same prompt took about 3.3 seconds longer on the replica without the cache.
• What a managed service takes over. See the effort comparison for Foundry managed compute.

Step 5.1: Create a cluster and a GPU node pool

export CLUSTER=aks-demo-manual

az aks create \
  -g $RG -n $CLUSTER --location $LOC \
  --node-count 1 --node-vm-size Standard_D4s_v6 \
  --generate-ssh-keys

az aks nodepool add \
  -g $RG --cluster-name $CLUSTER -n gpu \
  --mode User \
  --node-count 1 --node-vm-size $GPU_VM \
  --node-taints sku=gpu:NoSchedule --labels sku=gpu

az aks get-credentials -g $RG -n $CLUSTER --overwrite-existing
kubectl config use-context $CLUSTER
kubectl get nodes -o wide

 

Step 5.2: Install the NVIDIA device plugin

AKS installs the GPU driver on these GPU nodes, but the driver alone doesn't tell Kubernetes which GPUs are available for pods. This step deploys the open-source NVIDIA Kubernetes device plugin, which implements Kubernetes' device plugin framework.

Compared with a CPU-only cluster: Kubernetes already discovers and accounts for standard resources such as CPU and memory, so ordinary pods don't need a vendor device plugin. NVIDIA GPUs are specialized hardware exposed as the extended resource nvidia.com/gpu; Kubernetes needs the plugin to discover and allocate them. This cluster has both a CPU system pool and a GPU user pool. CPU-only workloads still run normally without the plugin, but adding a GPU VM doesn't by itself make its GPU schedulable. The sku=gpu label selects nodes and the taint restricts placement; neither registers GPU capacity.

The DaemonSet runs one plugin pod on each node labeled sku=gpu, including new matching nodes added later. Its tolerations let it run on the tainted GPU pool. Through the mounted /var/lib/kubelet/device-plugins directory, the plugin registers with each node's kubelet, discovers GPUs, and reports their availability and health. The kubelet then advertises nvidia.com/gpu as a node resource so the scheduler can place GPU-requesting pods.

When the vLLM pod in the next step requests a GPU, the kubelet uses the plugin to allocate the device and supply the device-access configuration needed by the container runtime. This DaemonSet doesn't install drivers or serve a model; it connects the node's NVIDIA GPUs to Kubernetes resource allocation. Without it (or an equivalent GPU resource integration), the vLLM pod would remain pending because its nvidia.com/gpu request couldn't be satisfied.

On this lab's NVIDIA T4, the next step's nvidia.com/gpu: 1 allocates a whole GPU to the container, not a fraction.   This demo deliberately keeps whole-GPU allocation and doesn't enable MIG or DRA (read Running more with less: Multi-instance GPU (MIG) with Dynamic Resource Allocation (DRA) on AKS | AKS Engineering Blog)

cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels: { name: nvidia-device-plugin-ds }
  updateStrategy: { type: RollingUpdate }
  template:
    metadata:
      labels: { name: nvidia-device-plugin-ds }
    spec:
      tolerations:
        - { key: sku, operator: Equal, value: gpu, effect: NoSchedule }
        - { key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }
      nodeSelector: { sku: gpu }
      priorityClassName: system-node-critical
      containers:
        - name: nvidia-device-plugin-ctr
          image: nvcr.io/nvidia/k8s-device-plugin:v0.16.2
          securityContext:
            allowPrivilegeEscalation: false
            capabilities: { drop: ["ALL"] }
          volumeMounts:
            - { name: device-plugin, mountPath: /var/lib/kubelet/device-plugins }
      volumes:
        - name: device-plugin
          hostPath: { path: /var/lib/kubelet/device-plugins }
EOF

# Expect: 1
kubectl get node -l sku=gpu -o jsonpath='{.items[0].status.allocatable.nvidia\.com/gpu}{"\n"}'

 

Step 5.3: Deploy vLLM with a Deployment and Service

cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: phi-vllm
spec:
  replicas: 1
  selector:
    matchLabels: { app: phi-vllm }
  template:
    metadata:
      labels: { app: phi-vllm }
    spec:
      nodeSelector: { sku: gpu }
      tolerations:
        - { key: sku, operator: Equal, value: gpu, effect: NoSchedule }
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args:
            - "--model=microsoft/Phi-3.5-mini-instruct"
            - "--served-model-name=phi-3.5-mini"
            - "--dtype=half"
            - "--max-model-len=4096"
            - "--gpu-memory-utilization=0.9"
            - "--trust-remote-code"
          ports:
            - { containerPort: 8000 }
          resources:
            limits: { nvidia.com/gpu: 1 }
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            initialDelaySeconds: 60
            periodSeconds: 10
            failureThreshold: 30
---
apiVersion: v1
kind: Service
metadata:
  name: phi-vllm
spec:
  selector: { app: phi-vllm }
  ports:
    - { port: 80, targetPort: 8000 }
EOF

kubectl rollout status deploy/phi-vllm --timeout=900s

 

Step 5.4: Call it

Setup port-forwarding from a different terminal    kubectl port-forward svc/phi-vllm 8000:80  

export OPENAI_BASE_URL="http://localhost:8000/v1"
export OPENAI_API_KEY=not-needed
export OPENAI_MODEL=phi-3.5-mini

python3 chat.py

What it cost you: a cluster, a GPU node pool, a device plugin, and a hand-written Deployment, Service, and probe, plus every future upgrade of each.

What you got: total control. You can swap the GPU, pin versions, change any engine flag, cache weights on a PVC, or serve a quantized or fine-tuned artifact.

 

 

Step 5.5: Multiple replica, one service: where ordinary load balancing falls short 

Everything discussed above assumes that a single endpoint serves the model. In practice, however, you will often run multiple vLLM pods to provide additional capacity and improve availability.

In this post, I’ll briefly explain the challenges that arise in this setup and share some results from my testing. A deeper exploration is beyond the scope of this post, so I plan to cover the architecture, behavior, and test findings in more detail in a future article.

 

The problem. Each vLLM pod keeps its own KV cache: the attention state it already computed for prompts it has processed. With automatic prefix caching, a pod that has seen a prompt's prefix can skip recomputing it during prefill. That cache lives in one pod's GPU memory and isn't shared.  A standard Kubernetes Service balances connections, not conversations or cache contents, so two requests that share a long prefix, such as consecutive turns of one chat, can land on different pods. One pod answers from its cache; the other computes the whole prompt from scratch. Both answers are correct. The difference is how much work each pod did.

I ran two replicas of the same model (Phi-3.5-mini on vLLM 0.31.0, one NVIDIA T4 each, prefix caching on). I primed one pod with a prompt of about 2,770 tokens, then sent the identical prompt to that warm pod and to the other, cold pod. The warm pod reused 2,768 of the 2,774 prompt tokens from its KV cache, and its time to first token (TTFT) was about 320 ms as the client saw it (about 73 ms measured inside vLLM). The cold pod had to compute all 2,774 tokens, and its TTFT was about 4.0 seconds (about 3.45 seconds inside vLLM).

A standard Kubernetes Service doesn't know which pod holds a prompt's KV cache, so requests that share a prefix can land on the pod that has to recompute it. That is why cache-aware routing matters: sending a request to the replica that already holds its prefix, while still balancing load, avoids seconds of repeated work for prompts this long on this GPU.

 

Cleanup

# Option 2: Container Apps (environment, app, and its Log Analytics workspace live in $RG)
az containerapp delete -g $RG -n $ACA_APP --yes
az containerapp env delete -g $RG -n $ACA_ENV --yes

# Option 3: delete the exact managed compute deployment and wait for completion.
# Reuse SUB, MC_RG, MC_ACCOUNT, and MC_DEPLOYMENT from Step 3.1.
python3 - <<'PY'
import os
from azure.identity import DefaultAzureCredential
from azure.mgmt.cognitiveservices import CognitiveServicesManagementClient

client = CognitiveServicesManagementClient(DefaultAzureCredential(), os.environ["SUB"])
client.managed_compute_deployments.begin_delete(
    resource_group_name=os.environ["MC_RG"],
    account_name=os.environ["MC_ACCOUNT"],
    deployment_name=os.environ["MC_DEPLOYMENT"],
).result()
print("Managed compute deployment deleted.")
PY

# Options 4 and 5: free the GPU nodes first
kubectl --context $CLUSTER   delete deploy phi-vllm --ignore-not-found
kubectl --context $CLUSTER_K delete workspace workspace-phi --ignore-not-found

# Everything else in the demo resource group (clusters, Foundry resource, Container Apps)
az group delete -n $RG --yes --no-wait

 

References

• Deployment overview for Microsoft Foundry Models
• Managed compute in Microsoft Foundry
• Deploy open-source models with managed compute
• Announcing Foundry Managed Compute: Run open models in Microsoft Foundry
• Fireworks models on Microsoft Foundry
• Fireworks on Microsoft Foundry privacy and compliance FAQ
• Using serverless GPUs in Azure Container Apps
• Azure Container Apps pricing