Blog Post

Azure High Performance Computing (HPC) Blog
9 MIN READ

Confidential AI on AKS: Running Phi-4-mini with KAITO and Azure NCC H100 GPUs

KentAltena12's avatar
KentAltena12
Icon for Microsoft rankMicrosoft
Nov 17, 2026

Deployment lessons, runtime compatibility, and performance observations from a single-GPU evaluation & benchmark

Why confidential GPUs matter

For organizations working with sensitive data, securing AI means more than encrypting storage and network traffic. During inference, applications must process prompts, model weights, intermediate calculations, and generated tokens in memory.

Confidential computing addresses this data-in-use protection requirement. Azure’s NCCads_H100_v5 series combines a confidential CPU virtual machine with an NVIDIA H100 GPU. The platform extends the trusted execution environment across the confidential VM and attached GPU, enabling protected GPU offload of data, models, and computation.

The Standard_NCC40ads_H100_v5 configuration provides:

ResourceSpecification
GPUOne NVIDIA H100 NVL with 94 GB GPU memory
CPU40 non-multithreaded AMD EPYC Genoa cores
System memory320 GiB
Local temporary storage800 GiB

This makes NCC an interesting platform for private inference, GPU-accelerated analytics, and other workloads where organizations want both GPU performance and stronger isolation of sensitive computation. However, confidential hardware is only one part of the solution. The operating system, signed GPU driver, inference engine, attestation policy, and application architecture must work together.  These Virtual Machines have the following features:

  • Trusted Execution Environment (TEE) that spans confidential VM on the CPU and attached GPU, enabling secure offload of data, models and computation to the GPU.  
  • VM memory encryption using hardware-generated encryption keys.  
  • Encrypted communication over PCIe between confidential VM and GPU.
  • Attestation: Ability for CPU and GPU to generate remotely verifiable attestation reports capturing CPU and GPU security critical hardware and firmware configuration.

 

 

Our evaluation explored that integration using Azure Kubernetes Service (AKS), the Kubernetes AI Toolchain Operator (KAITO), and Microsoft Phi-4-mini-instruct.

The outcome was encouraging: after addressing deployment and runtime compatibility issues, the NCC configuration completed 240 measured inference requests without failures, including concurrency eight.

Scope: This was a functional and performance evaluation—not a security certification. We did not validate CPU/GPU attestation, attestation-gated key release, or cryptographic assurance of result protection.

The deployment approach

We separated infrastructure ownership from model orchestration:

  • AKS managed the cluster and explicitly created GPU node pool.
  • NVIDIA GPU Operator managed the driver, container toolkit, device plugin, and GPU discovery.
  • KAITO managed the inference workload using existing GPU capacity.
  • vLLM served the model through an OpenAI-compatible API.
  • A separate CPU-only client pod generated benchmark traffic inside the cluster.

An important distinction: this evaluation used the upstream KAITO Helm deployment, not the AKS-managed AI toolchain operator add-on. Configuration options and release timing can differ between those deployment paths.

Challenge 1: AKS networking and the NCC SKU

Accelerated Networking is not supported

The NCC SKU documentation explicitly lists Accelerated Networking as unsupported.

That matters because an AKS node pool ultimately relies on an Azure Virtual Machine Scale Set. If its network configuration enables Accelerated Networking, Azure rejects the NCC allocation. We encountered this during node-pool lifecycle testing. An initial pool could be created successfully, but a subsequent scale-from-zero operation generated an incompatible networking configuration.

The practical lesson is:  A successful initial deployment does not prove that the pool’s complete scale-down/scale-up lifecycle works correctly.

What worked in this environment

For our dedicated test pool, the successful recovery was to delete the zero-count pool through the AKS API and recreate it at one node.

The recreated pool had the required primary network-interface setting:

enableAcceleratedNetworking: false

We inspected both the scale-set configuration and allocated network interfaces rather than relying only on the node-pool provisioning status.

A representative creation command was:

az aks nodepool add \ --resource-group "$RESOURCE_GROUP" \ --cluster-name "$CLUSTER_NAME" \ --name nccgpu \ --mode User \ --node-count 1 \ --node-vm-size Standard_NCC40ads_H100_v5 \ --os-sku Ubuntu2404 \ --node-osdisk-type Managed \ --node-osdisk-size 256 \ --gpu-driver none \ --node-taints sku=gpu:NoSchedule \ --labels apps=llm-inference

This command reflects the tested configuration; it is not a guarantee that every AKS release or regional deployment will behave identically.

Do not directly modify AKS-managed scale sets or NICs to work around the issue. Use AKS-supported operations and engage Azure support when the generated configuration is incompatible. Deleting a node pool is disruptive and should not be treated as a general production scaling strategy.  

Challenge 2: Bringing existing GPUs to KAITO

KAITO can provision GPU capacity automatically, but that was not the right fit for this evaluation. AKS documentation lists confidential GPUs as unsupported with Node Auto-Provisioning. We also needed explicit control over the NCC node pool, operating system, driver installation, and networking validation.  We therefore used KAITO’s bring-your-own-GPU nodes approach.

Here, “bring your own” means supplying existing, correctly configured Kubernetes GPU nodes. Those nodes can still be Azure-managed AKS node-pool instances.

Assign clear ownership

Our configuration gave NVIDIA GPU Operator ownership of the GPU software stack and disabled overlapping KAITO components.

The key KAITO Helm values were:

nodeProvisioner: byo featureGates: disableNodeAutoProvisioning: true nvidiaDevicePlugin: enabled: false gpu-feature-discovery: nfd: enabled: false gfd: enabled: false localCSIDriver: useLocalCSIDriver: false nodeSelector: kubernetes.azure.com/agentpool: system

Disabling KAITO’s device plugin did not eliminate the device-plugin requirement. NVIDIA GPU Operator supplied it instead.

The node still had to advertise:

nvidia.com/gpu: 1

We used a node label to identify inference capacity:

resource: labelSelector: matchLabels: apps: llm-inference

The custom inference pod template also selected the dedicated NCC pool and requested one GPU. For model storage, this evaluation used an ephemeral emptyDir volume rather than KAITO’s local CSI storage path. That simplified the test deployment, but model downloads and caches were lost when the pod or node was replaced. Production deployments should deliberately select and secure their storage strategy.

Challenge 3: The operating system and GPU driver must match

A generic GPU-node recipe was not sufficient.

Our initial Ubuntu 22.04 attempt encountered a confidential-VM compatibility error. The successful NCC configuration used Ubuntu 24.04 and an Azure confidential-VM kernel.

For the measured run, the relevant versions were:

ComponentTested configuration
AKS/Kubernetes1.34.12
GPU-node OSUbuntu 24.04.5 LTS
GPU-node kernel6.8.0-1071-azure-fde
NVIDIA GPU Operator25.3.1
NVIDIA driver580.178.04
KAITO controller0.13.0

We created the node pool with --gpu-driver none, then installed the GPU software through NVIDIA GPU Operator.

The driver image selection followed the node’s exact kernel. We also checked the loaded kernel module’s signer, rather than assuming an image name proved that the running module was signed.

Qualification included:

  1. Confirming the node was ready and used the expected OS/kernel.
  2. Checking nvidia-smi and the loaded module’s signer.
  3. Inspecting confidential-computing mode and GPU readiness.
  4. Waiting for NVIDIA operator validation.
  5. Running a CUDA vector-add smoke test.

A visible GPU is not necessarily a usable GPU. Passing an actual CUDA operation provided a stronger functional checkpoint than driver discovery alone.

Challenge 4: The initial vLLM runtime was not sufficient

The initial KAITO-provided serving configuration used vLLM 0.30.0.

That stack could load Phi-4-mini and return responses on NCC. However, the performance evaluation exposed problems that a basic smoke test did not reveal. For short requests, the original NCC run measured approximately:

ConcurrencyOriginal NCC output throughput
124.7 tokens/second (~1/10th non confidential performance)
4113.1 tokens/second (~1/10th non confidential performance)
8Failed

At concurrency eight, two allocations encountered CUDA illegal-memory-access errors and GPU Xid 31/MMU faults. Both recreated allocations received the same physical GPU, limiting our ability to separate device-specific behavior from software behavior.  Those results established that the original configuration was unsuitable for our benchmark. They did not establish the normal performance overhead of confidential computing, which is roughly 5%.

Why the CC-aware runtime changes matter

Upstream vLLM PR #59981 describes changes for the V2 GPU model runner under NVIDIA Confidential Computing.

The changes address two important behaviors:

  • Assumptions about directly accessing pinned host buffers through Unified Virtual Addressing can be invalid under CC.
  • CPU–GPU copy behavior can introduce host-side stalls that interfere with scheduling and decoding.

The fix detects confidential-computing mode, avoids the affected UVA path, stages host-to-device transfers, and moves device-to-host output copies onto a worker thread. These details explain why “CUDA works” and “the inference runtime is CC-ready” are different milestones.

What we upgraded

The requested minimum was vLLM 0.34.0rc3 or later containing the CC fix.  At preflight, release artifacts were not uniformly available across distribution channels. We located an official wheel associated with the v0.34.0 source tag whose package metadata reported 0.34.0, and verified that its source contained the relevant fix. The tested deployment used a digest-pinned official vLLM 0.33 base image and installed the hash-verified 0.34.0 wheel before launching inference.

The upgrade involved more than one package

The newer runtime also required dependency reconciliation. We removed unused optional packages that conflicted with the upgraded stack, updated OpenTelemetry packages, and recorded an intentional NCCL override inherited from the official base image.

A FlashInfer global-timer kernel failed to build because a development header was unavailable. Serving continued using a CUDA-event timing fallback, with a warning that CC-mode autotuning could be affected. We retained that warning as a qualification caveat rather than presenting the environment as warning-free. For a repeatable deployment, build and qualify an immutable image containing the complete tested dependency set instead of installing packages during every pod startup.

Testing environment

We evaluated a single NCC GPU against a retained non-confidential NC reference.

SettingNCC evaluationNC reference
VM sizeStandard_NCC40ads_H100_v5Standard_NC40ads_H100_v5
RegionEast US 2Spain Central
Measurement dateNovember 16, 2026November 13, 2026
GPUs per inference pod11
ModelPhi-4-mini-instructPhi-4-mini-instruct
vLLM0.34.00.30.0
Serving deploymentCustom KAITO template/native vLLMKAITO preset runtime
Client placementIn-cluster CPU system nodeIn-cluster CPU system node

Both deployments used the same pinned model revision:

cfbefacb99257ffa30c83adab238a50856ac3083

The principal serving settings were BF16 precision, a 4,096-token context limit, tensor parallelism of one, a maximum of eight sequences, and a GPU-memory-utilization target of 70%. Prefix caching and chunked prefill were enabled.

The inference service used ClusterIP connectivity. This removed an external internet path from the benchmark, but did not make the entire application confidential: the ordinary CPU client received generated output outside the confidential GPU node’s protection boundary. The internal HTTP benchmark also did not validate TLS protection.

Functional validation and benchmark methodology

We separated qualification into distinct layers.

LayerEvidence
InfrastructureExpected NCC allocation, compatible NIC configuration, ready node
GPU softwareSigned loaded driver, operator validation, successful CUDA vector add
RuntimeInstalled vLLM version/source evidence and active CC-aware copy-path logs
Model servingModel loading, health endpoint, coherent-response smoke test
PerformanceCompleted streamed requests with exact reported token counts

For the performance phase, we used two synthetic request sizes:

CaseInput tokensOutput tokens
Short128128
Long2,048512

Each case ran at concurrency 1, 4, and 8, with 40 measured requests per combination: 240 requests per deployment. Warmups were excluded.

The client used closed-loop fixed concurrency and streaming responses. It recorded request latency, time to first nonempty content, token usage, failures, and output throughput.  Throughput was calculated as:

Successful output tokens / measured case wall-clock duration

Performance results

Output throughput

WorkloadConcurrencyNC reference, vLLM 0.30NCC, vLLM 0.34NCC vs. reference
Short1265.6 tok/s266.9 tok/s+0.5%
Short41,017.8 tok/s1,008.1 tok/s−1.0%
Short81,988.4 tok/s2,000.8 tok/s+0.6%
Long1264.8 tok/s271.2 tok/s+2.4%
Long4942.4 tok/s956.8 tok/s+1.5%
Long81,623.9 tok/s1,640.7 tok/s+1.0%

NCC latency

WorkloadConcurrencyMedian request latencyP95 request latencyMedian time to first token
Short10.478 s0.485 s34.3 ms
Short40.506 s0.526 s53.8 ms
Short80.512 s0.520 s53.7 ms
Long11.888 s1.891 s48.1 ms
Long42.140 s2.144 s145.2 ms
Long82.497 s2.503 s213.2 ms

The upgraded NCC configuration completed the concurrency-eight cases that had failed with the earlier stack. Its measured throughput was close to the retained NC reference across this workload matrix.

Operational lessons

The deployment exposed several useful boundaries between “working” and “ready”:

Validate lifecycle operations, not just creation. Initial provisioning succeeded, while scale-from-zero exposed an incompatible networking configuration.

Keep infrastructure and GPU-software ownership explicit. AKS managed the pool, NVIDIA GPU Operator managed the GPU stack, and KAITO orchestrated the model on existing capacity.

Qualify the complete runtime. Driver visibility and a CUDA smoke test did not predict the behavior of the original inference engine under concurrency.

Preserve results outside the worker’s lifetime. The orchestration worker ended before completing local result collection and cleanup. We recovered measurements from the client pod, preserved them, and returned the GPU pool to zero.

Verify cost cleanup independently. We confirmed zero GPU-pool capacity, zero scale-set capacity, and no remaining GPU instances. Retained CPU nodes and ancillary resources can still incur charges.

Conclusion

Azure NCC H100 GPUs offer a compelling foundation for AI workloads that need hardware-backed protection of data in use.  Our evaluation demonstrated a practical integration path with AKS and KAITO: explicitly provision the confidential GPU pool, verify compatible networking, install the matching signed driver stack, use KAITO’s BYO-node configuration, and qualify a CC-aware inference runtime.

After upgrading vLLM, the deployment completed 240 measured requests without failures, with throughput close to the retained NC reference.  That is a meaningful functional and performance result—but it is not the final step in a confidential-AI deployment. Attestation, policy-controlled release of sensitive assets, secure application transport, storage protection, and the handling of outputs remain separate production requirements.

References

Updated Nov 16, 2026
Version 1.0
No CommentsBe the first to comment