Azure High Performance Computing (HPC) Blog articles

Connecting Microsoft Discovery App to Azure HPC with Azure NetApp Files and CycleCloud

richpaw — Wed, 22 Jul 2026 16:47:17 GMT

Introduction

Traditional High-Performance Computing (HPC) is central to most scientific computing and engineering applications. These systems are based on well tested, repeatable, and scalable compute processes that enable the large-scale generation and validation that the semiconductor and other industries depend on.

Integrating AI and agentic flows into this traditional model remains a challenge. Where many agentic systems rely on “modern” compute architecture utilizing REST based communications and object storage, traditional HPC relies on POSIX based file systems and scheduler-based orchestration. Combining these two different disciplines presents a challenge to many engineering and large-scale research enterprises.

This blog describes how Microsoft Discovery app can directly interoperate with existing traditional HPC deployments utilizing Azure HPC. Microsoft Discovery is an enterprise agentic AI platform for research and development, designed to help specialized agents reason, plan, execute, and learn in a continuous loop across data, tools, and workflows. It is built on Azure and designed to integrate with capabilities such as Azure HPC and Microsoft Foundry, while leveraging industry proven tools and customer flows, making it a natural bridge between AI-native reasoning and compute-intensive engineering execution.

Agentic AI in HPC

The value of agentic AI in scientific and engineering workloads is not just as a chatbot or code generator; it is the ability to participate in an existing workflow: write parameter files, launch simulations, inspect logs, summarize failures, generate follow-on jobs, and preserve artifacts for traceability. To do this effectively, agents must be able also read data from POSIX-based file systems and dispatch jobs to the scheduler.

To maintain IP security, they must access data while respecting the access limits of the user that dispatched them and write data under the same RBAC rules as that same user. That way organizations can be assured that agents operate strictly under the dispatching user's access rights, so they cannot reach data the user isn't authorized to see and the data they write retains the permissions that the user who dispatches the agent enjoys.

Discovery app is currently only available on Windows. Linux and Mac versions will soon be available. In the meantime, enabling the Windows based Discovery app to leverage the capabilities on Azure HPC today with little to no change to the current HPC flow.

Reference architecture

The architecture has five primary components: a Windows VM running the Discovery app on Azure, an Azure NetApp Files volume that is mounted by both Windows and Linux clients, an Azure CycleCloud-managed HPC cluster with a Linux login node used for job submission and monitoring, and a scheduler that dispatches work to compute nodes.

Discovery app on Windows: Runs on a Windows VM placed inside the same VNet, or reachable through controlled private networking, so it can access storage and the login node without exposing the HPC environment publicly.
Azure NetApp Files: Provides the shared file namespace for input decks, scripts, logs, generated data, and results. Linux nodes mount the volume with native NFS. Windows can mount the same namespace through the Windows NFS client or an equivalent enterprise file-access solution.
Azure CycleCloud: Provisions and manages the HPC cluster, including scheduler, login node, execute nodes, autoscaling, and cluster configuration.
Login node: Acts as the controlled command endpoint. Discovery should submit jobs and query scheduler state from here, not run heavy tools directly on the login node. This protects the actual Cyclecloud scheduler and cluster manager from being overwhelmed if the dispatch volume is high. This also allows for scalability if there are many clients scheduling jobs at once.
Scheduler: Incumbent schedulers that customers are already using with Azure Cyclecloud handles execution. Discovery creates scripts and submits them through the scheduler so compute runs on the correct nodes with the correct environment, licenses, and policies. This remains the same as how traditional HPC schedules jobs today.

Connectivity model

Running on a Windows VM in the Azure VNet aligns with many enterprises’ requirement that none of their data or IP can leave their network. Doing this avoids the need to run jobs through a user laptop or unmanaged public endpoint. This also allows the user to start an agentic workload then disconnect from the VM while the job runs and reconnect when they come back. Maintaining the same working model most engineers have with their HPC environment today.

For Discovery to access Azure HPC, it needs to map the Azure NetApp Files volumes and command access to the Linux login node. File access lets agents create and read files. Command access lets agents run Linux commands, submit jobs, and monitor execution. Discovery writes execution scripts to the ANF volume, then uses SSH to dispatch the job to the schedule with the execution script.

The advantage of this model is that the user can directly provide Unix-style paths to Discovery and tell it to execute tools without the user worrying about being on a Windows platform. For example, CAD teams often provide setup scripts with paths to tools that users are supposed to reference in creating their own tool execution scripts. Using this model, I just tell Discovery to use the setup script /mount/setup/tools/vendora.csh as a template and it will happily find the file and do so. I can then tell Discovery to dispatch the job to the hpc queue and it will schedule to the appropriate SLURM queue where a system is provisioned and assigned to run using the new execution script Discovery creates.

This model also gets around the temporary limitation of Discovery app, unlike the enterprise Discovery service, only being available on Windows. In this model, Windows is only acting as a terminal interface so it doesn’t matter that it’s not a platform that normally works with most EDA tools. In fact, users can deploy Discovery app on either x86 or ARM Windows since all the actual tools run on the Linux-based HPC deployment in Azure anyway.

Mapping Windows drive to NFS volumes

Discovery agents must understand how to access files on the mapped NFS volume from the Windows VM. The same file may appear as Z:\project\run1\exec.csh on Windows and /mount/project/run1/exec.csh on Linux. The skill or instruction file used by Discovery should explicitly describe that mapping and instruct agents to use Linux paths inside batch scripts and scheduler commands.

A typical instruction might say: the Windows drive Z: maps to the Linux mount point /mount. If Discovery sees /mount/designs/test.sv, it can read or write the corresponding Windows file at Z:\designs\test.sv. When generating job scripts, it should always use the Linux path, because those scripts run on the cluster.

Implementation steps

1. Deploy the HPC foundation

Start with the standard Azure HPC foundation: a virtual network, Azure CycleCloud, a scheduler-backed cluster, and shared storage. CycleCloud can deploy and manage Slurm clusters and supports external NFS mounts, including Azure NetApp Files. Configure the cluster so the scheduler and compute nodes mount the shared file system at a stable Linux path such as /mount, /shared, or a customer-specific project path.

Make sure that systems can ssh to the login node using a public/private key pair. The login node should be accessible using a command like:

% ssh -i .ssh/keyfile user@login-node

2. Place the Windows VM in the same network boundary

Create a Windows VM that can reach both the Azure NetApp Files endpoint and the CycleCloud login node over private IP. The required Windows NFS client is available on Windows 11 Pro, Enterprise, and Education. It is not available on Windows 11 Home. Install the Discovery app on that VM. This VM becomes the user’s Discovery workstation for the HPC-connected workflow. Keep the VM inside the same security boundary as the HPC deployment, and use standard enterprise controls such as private networking, restricted inbound access, managed identity where appropriate, and least-privilege SSH keys.

3. Install Microsoft Discovery app

Download and install Microsoft Discovery app from:
https://github.com/microsoft/discovery
Once installed, start Microsoft Discovery app and connect to your Github Copilot account in order to configure access to the LLM AI models.
Create a new project

4. Mount the Azure NetApp Files volume into Windows

Mount the same shared storage namespace into Windows. The Windows NFS client can map the NFS export to a drive letter such as Z:. The Windows NFS is sufficient for reading logs, results, and data files, and writing execution scripts, parameters, and configuration files. Heavy simulation I/O remain on Linux compute nodes running HPC tools.

To configure the Windows NFS Client, you will need to have administrator privileges on your Windows VM.

To mount the ANF volume with the Windows NFS client.

Press Win+R
Run “optional features”
Select “More Windows features”
Expand “Services for NFS”
Enable “Administrative Tools” and “Client for NFS”
Click “OK” to install the components.
By default, Windows will mount the NFS volume as an anonymous user. To set the correct user, you can define the correct Linux User ID and Group ID in the registry.

Note: This is an example of a functional baseline to mount a Linux volume with a user identity as an example. Production environments should employ proper identity mapping for security.
Press Win+R
Run “regedit” to start the Registry Editor
Navigate to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ClientForNFS\CurrentVersion\Default
In Default, right click and add two new DWORD entires: AnonymousUID and AnonymousGID
Set the values to the decimal value of your Linux UID and GID. This will ensure that you connect to the ANF volume as the correct user.
Select OK and then launch the Command Prompt as Administrator.
To allow the NFS Client to accept the new registry entries, restart the NFS Client with the following commands:
nfsadmin client stop

nfsadmin client start

Mount the ANF volume to a Windows drive:
You should now be able to access your NFS volume by going through the Z: drive.

5. Configure SSH access to the login node

Configure the SSH client on the Windows VM with a dedicated key for Discovery-driven access. The key should connect to the CycleCloud login node as the user (same as with NFS above) with permissions to submit jobs, read scheduler state, and access the shared project directory. Verify connectivity by connecting to the Cyclecloud login node using ssh from the Command Prompt.

In this example, the username is ai4semi, the IP address of the login node is 10.16.16.25, and the cc_rsa is the keyfile located under the .ssh directory of the user’s home directory on Windows. Since this is the first time for this system to log into the Linux machine, you must accept the connection registration. This will not be necessary for subsequent ssh connects by Discovery using this method.

5. Teach Discovery how to use Azure HPC

Now that you have the NFS and SSH connections established in Windows, you simply need to tell Discovery how to use these connections properly to run HPC workloads. In the Discovery chat window, tell Discovery to map the Z: drive on windows to the /mount volume on NFS and how to interact with the files.

The Windows drive `Z:` is mapped to the cluster NFS volume `/mount`.

- `/mount/<path>` on the cluster corresponds to `z:\<path>` locally.

- Example: when a file is referenced as `/mount/test.txt`, look at `z:\test.txt`.

Test to make sure Discovery understand by asking it to read a file on the NFS volume using POSIX nomenclature.

read the contents of /mount/newfile.txt

Discovery should show you the file contents

Next, instruct Discovery on how to access and interact with Cyclecloud

Connect to the CycleCloud login node over SSH using "ssh -i .ssh\cc_rsa ai4semi@10.16.16.25"

Then tell Discovery which scheduler Cyclecloud uses and tell it to check the queue.

Cyclecloud is using SLURM as the scheduler. Check the status of the hpc queue.

Since the Windows VM is only used to run Discovery and the agentic AI part of the workload and the login node is not meant to run actual HPC workloads, be explicit with Discovery on how it should treat HPC job runs.

NEVER run EDA/build/simulation tools on the local system or directly on the login node. ALWAYS submit work to the Slurm scheduler via sbatch.

- The login node is for job submission and monitoring only — not for compute.

- Wrap every tool invocation in a batch script and submit it with sbatch.

6. Validate the end-to-end workflow

Validate that everything works by dispatching a simple test script to Cyclecloud

Create a batch script that references the Linux path, submit it to Cyclecloud.

dispatch /mount/proj/ai4semi/exec.csh to the hpc queue

7. Save the knowledge for future reference

Now that Discovery understands how to access Linux-based files on NFS and how to dispatch jobs, tell it to save the information.

Save this knowledge to a skill and use it for all subsequent job runs and file access for the project

You see that Discovery now saved the information on how to access files and run jobs to a SKILL.md for general knowledge and updated the copilot-instructions.md to ensure that all subsequent agents and engines understand how to access Azure HPC.

Note: These instructions are only scoped to a single project. When creating a new project, you can simply tell Discovery to reach over to your older project to copy the knowledge:

copy over instructions on how to access NFS files and paths, and how to dispatch and execute hpc jobs from workspace1

If you want to make this a portable instruction that you can just load for each new project, tell Discovery to create a portable file.

package the instructions on how to access NFS files and paths, and how to dispatch and execute hpc jobs into a a single instruction I can provide to new workspaces

You can then copy the file to a central location like: C:\Users\ai4semi\project

Then for each new project you can tell Discovery to use this file.

read and understand the instructions in C:\Users\ai4semi\project\cyclecloud-hpc.instructions.md. Use for all agents and engines in this project and save this information as skills to use in the project

Operational considerations

Identity and permissions: Keep SSH access scoped to the project and avoid broad administrative privileges. For NFS, verify UID/GID mapping and file ownership behavior before production use.
Performance: Use the Windows mount for orchestration artifacts, scripts, and logs. Let Linux compute nodes perform heavy I/O through native ANF mounts.
Security: Keep Discovery, storage, and the login node on private networking. Avoid public exposure of scheduler or storage endpoints.
Scheduler hygiene: Enforce a rule that Discovery submits jobs through the scheduler and does not run heavy tools interactively on the login node.

Conclusion

Connecting Microsoft Discovery app to Azure HPC allows users to seamlessly use their existing HPC environment. The most practical approach today is to place Discovery on a Windows VM inside the Azure network, map the shared Azure NetApp Files namespace into both Windows and Linux, and use SSH to submit scheduler-managed jobs through CycleCloud.

Using this method, customers can quickly and easily use Discovery to integrate Agentic AI into their current HPC flows. Users can take advantage of AI agents to read and understand specification files, generate code and testbenches based on those specifications, run existing HPC tools on their Azure HPC environment. They can direct Discovery to use the scripts and resources which their CAD teams have developed. Users can create bookshelves by pointing to the documentation directories many tool vendors include in their installations, allowing Discovery to take advantage of documentation which most users don’t have time to fully digest.

Over time, native Linux support can reduce the need for Windows-hosted bridging. Until then, this architecture gives teams a secure, and repeatable way to let Discovery agents create, launch, monitor, and learn from real HPC workloads on Azure.

Microsoft Discovery: Where HPC meets agentic AI for the next era of EDA

richpaw — Wed, 22 Jul 2026 15:25:57 GMT

Introduction

High-performance computing is the engine behind the most demanding engineering breakthroughs. In electronic design automation, HPC enables massive simulation farms, verification regressions, place-and-route exploration, timing closure, power analysis, and signoff workloads that would be impractical at scale. However, as chip complexity grows, the limiting factor is no longer only compute capacity. It is the ability to reason across enormous design spaces, coordinate specialized tools, preserve engineering context, and decide what to run next.

That is where Microsoft Discovery becomes especially interesting. Microsoft Discovery is an enterprise agentic AI platform for research and development, designed to help specialized agents reason, plan, execute, and learn in a continuous loop across data, tools, and workflows. It is built on Azure and designed to interoperate with capabilities such as Azure HPC and Microsoft Foundry, while leveraging industry proven tools and customer flows, making it a natural bridge between AI-native reasoning and compute-intensive engineering execution.

The Inflection Point: HPC is required, but not enough

Engineering teams already know how to scale compute. They run large regressions, distribute workloads across clusters, burst into cloud capacity, and optimize storage and schedulers around fast-moving design cycles. Yet the harder challenge is often deciding which simulations matter, which failures are related, which tool settings deserve another experiment, and which signals should trigger the next branch of exploration.

Traditional HPC gives engineering teams scale. Agentic AI adds accelerated analysis, orchestration, memory, reasoning, and adaptation. The synthesis of the two is not about replacing engineers or replacing EDA tools. It is about turbocharging the engineers by creating an intelligent execution fabric where agents can understand goals, inspect results, choose tools, launch jobs, summarize outcomes, and recommend the next best action. Engineers remain in control of the design intent and final decisions. In this model, the most valuable resource is not compute capacity; it is engineering time.

Microsoft Discovery as the agentic layer for engineering R&D

Microsoft Discovery focuses on the full R&D lifecycle: knowledge reasoning, code and plan generation, simulation, analysis, and iteration. For semiconductor and EDA workflows, that maps naturally to the way design teams already operate. Timing closure, regression failure, achieving performance targets all require that engineers pour over massive amounts of information from tool run logs, interpret the results based on engineering goals and specifications, and determine how to resolve the gaps. Discovery leverages the same scientific process used in scientific research to help engineers more quickly understand the resulting data, pick out critical learnings, and help compose the next steps and mitigations.

Unlike chat interfaces, agents can act and execute specific functions. Agents can be assigned roles: a verification triage agent, a log-analysis agent, a simulation-planning agent, a physical-design exploration agent, a cost-and-capacity agent, or a documentation agent that preserves evidence and rationale. Together, these agents can coordinate around a shared objective and use HPC as the execution and validation vehicle.

For semiconductor programs, engineers are the most expensive and constrained part of the process. Every hour spent searching logs, reconciling reports, relaunching routine jobs, copying context between systems, or documenting repetitive findings is an hour not spent on architecture, debug strategy, design tradeoffs, or design sign-off. Discovery’s value is therefore not simply that it can automate tasks; it can help shift engineering effort away from menial coordination work and toward the decisions that improve product outcomes and silicon spins.

What this looks like for EDA

In a complex silicon project, design teams may run thousands of tests across multiple scenarios, configurations, simulators, and coverage targets. Today, engineers spend significant time reading logs, correlating failures, rerunning jobs, and deciding whether a failure is new, known, flaky, or blocking. Much of that work is necessary, but repetitive. With an agentic workflow, a verification agent could monitor regression output, cluster related failures, inspect logs, identify likely root causes, and propose targeted reruns rather than brute-force rerunning everything. All this helps to provide engineers more time to focus on the work that requires their unique engineering skill and judgement.

In physical design, an agentic workflow could help explore constraints, placement strategies, timing violations, congestion hot spots, and power tradeoffs. HPC provides the parallel capacity to run experiments. Discovery-style agent orchestration can help determine which experiments to run, capture why they were run, compare outcomes, and refine the next set of candidates. This provides a guided design-space exploration loop.

For signoff and analysis, agents could help connect results across timing, power, reliability, manufacturability, and cost. Instead of treating each report as an isolated artifact, an agentic system can reason across reports, prior runs, known design patterns, and engineering guidance helping to converge faster.

Discovery leverages tried-and-true, industry-proven tools to validate the output of AI using the same methods semiconductor teams have relied on for decades. AI can propose code and plans, prioritize validation and simulation, summarize analysis and debug, or recommend next steps, but the validation path still runs through established EDA flows that engineers understand and trust.

The Architecture: Agents, Tools, Data, and Compute

Discovery’s flow architecture leverages a closed-loop system. Knowledge sources provide context: design specs, prior bugs, regression history, tool documentation, scripts, and engineering notes. Agents reason over that context and formulate next steps. Tool integrations connect those agents to EDA applications, schedulers, storage systems, model endpoints, and analysis pipelines. Azure HPC supplies scalable execution for simulations and compute-heavy analysis. Results flow back into the system so the next decision is better informed than the last.

This demonstrates the power of joining agentic AI and HPC. Discovery provides a platform model for coordinating that loop with enterprise expectations around security, governance, transparency, and human oversight. HPC tools provide assurance that the results from AI are valid.

Integrating Discovery with Azure HPC for EDA

Discovery becomes even more powerful when it is connected to Azure HPC infrastructure purpose-built for compute- and memory-intensive engineering workloads. For EDA, that includes AMD-based Azure HPC virtual machines such as HX-series instances, which are optimized for silicon design and memory-intensive workloads, with large memory capacity, AMD EPYC processors with 3D V-Cache, and high-performance InfiniBand networking. These platforms give agentic workflows the execution fabric needed to run large simulations, regressions, analysis jobs, and design-space exploration at cloud scale for both EDA and other HPC workloads.

Storage is equally important. EDA workloads generate and consume enormous numbers of files, and performance often depends on low-latency shared file access as much as raw CPU capacity. Azure NetApp Files provides enterprise-grade file storage for mission-critical HPC and EDA workloads in Azure, giving teams a managed, high-performance storage layer that can support simulation farms, tool installations, design libraries, scratch spaces, and shared project data.

Azure NetApp Files also enables customers to use Discovery in a hybrid environment, enabling customers to keep critical data on-prem while using Discovery to optimize cloud-based flows. Many semiconductor teams keep authoritative design data, source repositories, IP libraries, and sign-off environments in on-premises infrastructure. Azure NetApp Files cache volumes can help bridge that workflow by creating a cloud-based cache of active data from an external origin volume. Frequently accessed data can be served close to Azure compute, while the authoritative dataset remains in the existing on-premises design environment.

Cluster automation is another important part of enabling the hybrid flow. Azure CycleCloud can help automate the creation, scaling, and lifecycle management of HPC clusters in Azure so teams can stand up cloud capacity around specific EDA workloads rather than manually managing static infrastructure. That is important for agentic workflows because Discovery can reason about what needs to run, determine the proper resources needed, while CycleCloud-backed automation helps ensure the right compute environment is available when those jobs are ready to execute.

Just as importantly, hybrid adoption does not need to force customers to replace the scheduling systems their engineering organizations already use. Many semiconductor teams have deep operational investments in incumbent schedulers which CycleCloud already works with, established queues, policies, licenses, scripts, and user workflows. A practical Discovery-plus-Azure-HPC architecture helps bring agentic AI to their existing flows: integrating cloud capacity with existing scheduler patterns, extending familiar job-submission models, and allowing teams to burst selected workloads into Azure while preserving the operational controls that have governed EDA execution for years.

In that architecture, Discovery can act as the intelligent orchestration layer across the hybrid estate as well as a productivity accelerator. Agents can reason over design intent, prior results, and engineering context; submit jobs into Azure HPC environments; use AMD-powered compute for demanding EDA runs; access hot data through Azure NetApp Files; and work through cluster automation and scheduler integrations that align with customer operating models. Combined with Azure HPC, Azure NetApp Files helps eliminate storage bottlenecks that can otherwise limit EDA job throughput, allowing compute resources, EDA licenses, and engineering teams to remain productive at scale. The result is a practical synthesis: AI-guided exploration, proven EDA validation, elastic cloud compute, hybrid data access, and scheduler-aware execution that respects the way semiconductor teams already work.

Why it matters for semiconductor teams

Semiconductor design is a systems problem. Teams must manage more IP, more verification complexity, more software interaction, more foundry requirements, and more pressure to maintain schedules while design sizes and complexity grows. The industry has invested heavily in automation, but much of that automation remains fragmented across scripts, dashboards, job schedulers, and individual expert workflows. The simple fact is that compute is expensive, but engineering time is even more expensive. Improving engineer productivity has an outsized impact on schedule, quality, and cost.

Agentic AI offers a way to automation more of the design and verification process. Automating much of the more mechanical parts of the flow. Teams can create workflows that observe, reason, and iterate while agents coordinate the results to achieve the engineering objective. Instead of relying only on tribal knowledge, teams can preserve decisions, evidence, and rationale in a repeatable workflow.

Human-in-the-loop by design

The goal is not autonomous chip design without engineers. The goal is amplified engineering outcomes. In the most valuable scenarios, agents handle the repetitive, high-volume, evidence-gathering work while engineers define objectives, validate assumptions, approve major decisions, and interpret tradeoffs. Discovery can help reduce the menial burden of searching, summarizing, comparing, rerunning, and documenting so engineers can spend more time on the creative and analytical work only they can do. This is especially important in EDA, where correctness, traceability, and sign-off confidence matter as much as speed.

Microsoft Discovery’s emphasis on enterprise governance and transparency is therefore central to the story. In regulated or high-stakes engineering environments, teams need to know what data was used, which tools were invoked, what assumptions were made, and where human approval occurred. Just as important, they need confidence that AI-generated recommendations are validated through established engineering practices, not accepted on faith. Agentic workflows must be auditable, explainable, and grounded in the same EDA verification and signoff discipline the industry already depends on.

Conclusion: Going beyond just compute and AI

The next phase of EDA acceleration will not come from compute alone, and it will not come from AI alone. It will come from the synthesis of both: HPC infrastructure that can execute at scale, and agentic AI systems that can reason, coordinate, and learn across complex engineering workflows.

Microsoft Discovery steps towards that future. For HPC and EDA teams, the opportunity is to move from faster batch execution to intelligent convergence: workflows that know what has been tried, understand what changed, recommend what to try next, and use scalable compute to validate ideas quickly through proven EDA tools and methodologies. In a world where design complexity keeps rising, the ability to combine elastic compute with agentic systems can help to greatly improve productivity, turn repetitive process work into guided automation, and validate AI-assisted decisions through the same trusted engineering practices the semiconductor industry has used for decades.

The Microsoft Discovery team will be at Design Automation Conference, July 27-29, 2026, in Long Beach, California. The Microsoft Discovery team will be showcasing how engineers can use agentic AI to enable greater productivity for semiconductor design workloads. Drop by the Microsoft booth (651) to learn more about Microsoft Discovery for semiconductor design. Learn how technologies from AMD and NetApp come together with Azure HPC to provide the high-performance environment for running EDA tools on the cloud.

Teamcenter on Oracle AI Database@Azure: Architecture, Validation, and Results

Sunita_AZ0708 — Tue, 07 Jul 2026 21:41:05 GMT

Overview

Manufacturing organizations depend on Product Lifecycle Management (PLM) platforms to manage engineering data, product structures, digital twins, and collaborative design processes. As enterprises modernize their infrastructure, many continue to require advanced Oracle database capabilities such as Oracle RAC while simultaneously adopting Microsoft Azure for application hosting, AI, analytics, and digital engineering workloads.

Oracle Database@Azure brings Oracle Exadata capabilities directly into Azure datacenters through a tightly integrated multi-cloud architecture that combines Microsoft Azure and Oracle Cloud Infrastructure (OCI). This approach enables customers to run Siemens Teamcenter application workloads on Azure while leveraging Oracle Exadata Database Service for enterprise-class availability, scalability, and performance. Architecture

Teamcenter multi-tier architecture deployed on Azure with Oracle Exadata Database(OCI) Service.

Architecture

The solution uses a multi-tier Teamcenter deployment hosted on Azure Virtual Machines while leveraging Oracle Exadata Database Service on Dedicated Infrastructure.

Client Tier

Teamcenter Rich Client
Active Workspace Client
Azure Virtual Desktop
Engineering Workstations
Siemens NX

Web Tier

Teamcenter Web Tier
Active Workspace Gateway
Teamcenter Security Services

Enterprise Tier

Teamcenter Foundation
Active Workspace
Visualization Services
File Management System (FMS)
Apache Solr Search
Dispatcher Services
File Server Cache

Database Tier

Oracle Exadata Database Service
Oracle RAC
Autonomous Recovery Service
Backup Infrastructure

Key capabilities include:

Oracle Exadata X11M Infrastructure

2 Database Servers
3 Storage Servers
Oracle Database 19c
1,520 ECPUs
2,780 GB Memory
240 TB Total Storage
Oracle RAC

Applications execute natively on Azure while Oracle Exadata infrastructure is managed through OCI.

Benefits include:

Oracle feature parity
Oracle RAC support
Native Azure management experience
Integrated monitoring
High availability
Low-latency architecture

Identity and Security

Oracle Database@Azure supports integration between:

Microsoft Entra ID
Oracle Identity Cloud Service (IDCS)

Benefits include:

Single Sign-On
Centralized identity management
Federated authentication
RBAC governance

Benefits include:

No public database endpoints
Private IP communication
Encrypted traffic
Network Security Groups
OCI Security Lists

Deployment Requirements

Software Versions

Component	Version
Teamcenter Foundation	2506
Active Workspace	6.2
Deployment Center	14.2
Oracle Database	19c
Wildfly	26.1.x
Docker	24.x
OpenJDK	17
RHEL	8.x
Windows Server	2019

Teamcenter Solutions

The architecture supports:

NX Integration
Solid Edge Integration
Classification
Requirements Management
Schedule Manager
Change Management

Deployment Methodology

The validation followed a three-phase implementation model.

Phase 1 – Foundation

Oracle Database@Azure setup
OCI tenancy integration
Azure subscription mapping
Exadata provisioning
Teamcenter base installation

Phase 2 – Solution Deployment

Teamcenter Foundation
Active Workspace
Microservices
Solr
Dispatcher
Visualization Services
NX Integration

Phase 3 – Validation

Functional testing
Integration testing
Performance testing
Recovery validation
Production readiness assessment

Oracle Autonomous Recovery Service

Oracle Autonomous Recovery Service (ARS) was used for backup validation.

Capabilities include:

Scheduled backups
Azure or OCI backup storage

Observed backup behavior:

Daily automated backups
Autonomous Recovery Service destination
Successful recovery validation

Data Backup storage @ Azure

Backups stored in Azure Storage enable independent recovery options in the event of database corruption, accidental deletion, or infrastructure failures within OCI.
Azure-based backups provide an independent recovery path for business continuity.
The production database will reside on Oracle Cloud Infrastructure (OCI), while customer data backups will be securely stored in Azure Storage to ensure durability and data protection.

Select Azure storage as data backup storage location

Validation and Test Results

A comprehensive validation suite consisting of 23 certification tests was executed.

Key scenarios included:

User Experience

Active Workspace login
Rich Client login
Browser compatibility
External user access
Generic account access

Product Data Management

Item creation
Dataset attachment
Large file upload
Impact analysis
Release workflows

Engineering Workloads

NX Integration
JT generation
Check-in/Check-out
Visualization rendering

Platform Services

Dispatcher validation
Solr search
File Management Services
Pool Manager
Microservices

Recovery Operations

Backup creation
Restore testing
Database recovery verification

All validation scenarios are completed successfully.

Performance Analysis

Login Performance

Function	Average Time
Teamcenter RAC Login	~3 seconds
Active Workspace Login	~12 seconds
Solr Search	~1 second
Item Creation	~4 seconds
Dataset Attachment	~4 seconds

BOM Performance

Standard BOM Validation

Successful BOM report generation
Successful BOM navigation
Stable system operation
Large BOM Testing

Validation included:

6,000 Line-Item assemblies
12,000 Line items Bill of Material

Observed behavior:

Report Generation: ~80 minutes- ~160 minutes
Database Activity: 5–10 minutes

Database analysis indicated:

No database bottlenecks
All indexes valid
No deadlocks
No blocking locks
Fresh optimizer statistics

The investigation suggests most runtime occurred within Teamcenter application processing rather than Oracle database execution.

Performance Efficiency

Oracle Exadata Database Service is designed to support high-throughput transactional workloads and large enterprise databases. During validation testing, Teamcenter services successfully completed user authentication, product structure operations, search activities, workflow execution, and engineering data management functions. Oracle RAC and Exadata smart infrastructure provide a scalable foundation for increasing user populations and expanding product data volumes. Performance testing demonstrated that database operations remained efficient throughout validation activities, indicating that future optimization efforts should focus primarily on application-tier processing, workflow configuration, and Teamcenter customization strategies when addressing large-scale BOM workloads.

To minimize network latency between application and database tiers, the Teamcenter Enterprise Servers and Oracle Database@Azure infrastructure were deployed within the same AZ- Azure Availability Zone and co-located using white-glove based pinning, helping ensure low-latency communication for database-intensive transactions and PLM operations.

For production deployments, organizations should follow Siemens Teamcenter performance and sizing recommendations available through Siemens Support. Key optimization areas include Teamcenter pool manager configuration, FSC/FMS cache sizing, Solr indexing and search tuning, Active Workspace configuration, JVM heap sizing, database connection pool management, and workflow customization reviews. Regular monitoring of application response times, SQL execution patterns, and large BOM processing workloads can help identify opportunities for continuous performance optimization. Validation results indicated that Oracle Exadata Database Service was not a performance bottleneck; therefore, future tuning efforts are expected to provide the greatest benefit within the Teamcenter application and middle-tier services.

Cost Optimization

This article presents a baseline architecture for Oracle AI Database@Azure. Organizations can use this guidance as a starting point when planning and deploying their Development, Test, and Production environments, tailoring the architecture to meet their unique operational, performance, and compliance requirements.

Organizations should evaluate infrastructure sizing based on expected user populations, database growth rates, and workload characteristics.

Cost management recommendations include:

Right-size Azure Virtual Machines hosting Teamcenter services. Refer Teamcenter Baseline Architecture
Use autoscaling where applicable for supporting services.
Align Oracle Exadata capacity with actual workload requirements.
Leverage Azure credit where applicable.
Monitor storage growth and archive inactive engineering data.
Review backup retention policies regularly.
Consolidate non-production environments when possible.

While Oracle Database@Azure may represent a larger infrastructure investment than smaller standalone database deployments, it can reduce operational complexity, improve scalability, and eliminate the need for separate cloud connectivity architectures.

Conclusion

The validation effort demonstrated that Oracle Exadata Database Service on Azure successfully supports Siemens Teamcenter enterprise PLM workloads.

Key outcomes included:

Successful Teamcenter deployment on Azure
Oracle RAC support through Exadata X11M
Validated Azure + OCI architecture
Successful backup and recovery validation
Stable Teamcenter operation across all tested scenarios
No database-side tuning required during validation

For production deployments, additional application-level optimization is recommended for very large BOM processing workloads. Database performance remained healthy throughout testing and was not identified as the primary bottleneck.

Key Takeaways

Teamcenter 2506 successfully validated on Oracle Database@Azure.
Oracle RAC supported through Oracle Exadata Database Service.
Backup and recovery validated using Autonomous Recovery Service.
All certification tests completed successfully.
No database-side bottlenecks identified during testing.

References:

Overview - Oracle AI Database@Azure | Microsoft Learn

Onboard Oracle AI Database@Azure | Microsoft Learn

Exadata Database Service | Oracle

Siemens Teamcenter baseline architecture on Azure - Azure Architecture Center | Microsoft Learn

Inside Llama 3.1 405B MLPerf Training on Azure: System-Level Insights at 8K+ GPU Scale

shantanupatankar — Wed, 24 Jun 2026 16:13:35 GMT

By Shantanu Patankar, and Azin Heidarshenas

As part of Azure's MLPerf Training v6.0 submission, we scaled Llama 3.1 405B (NVFP4) pretraining to 8,192 GB200 GPUs across 128 racks on Azure's Fairwater infrastructure, converging in 7.07 minutes (MLPerf Training Blog). Llama 3.1, with 405 billion active parameters, has the highest active parameter count among all models in the MLPerf suite.

Getting a model of this scale to train efficiently requires a cascade of tightly coupled choices around parallelism, batch sizing, and network placement. This post shares what we learned from bringing up and scaling Llama 3.1 405b on Fairwater at frontier scale. Three insights emerged:

Even at 8K+ GPU scale, Llama 3.1 405B training remains compute-bound, with Feed forward Network (FFN) kernels dominating execution
Topology-aware mapping is the key enabler of 8000+ GPU scale
Scaling efficiency is governed by convergence dynamics, not system or network limits

The rest of this post explains how these effects appear in practice.

Where Does the Time Go Inside One Iteration, and What Are the Bottlenecks?

Llama 3.1 405B is a dense transformer with 126 layers, each consisting of a self-attention block (QKV projections and attention) followed by a feed-forward network (FFN). To understand what limits performance at scale, we profiled steady-state iterations and broke down where time is spent, as shown in Figure 1. A single training step (Global Batch Size=1024) takes approximately 1.27s.

Figure 1: Compute vs Communication Breakdown of Llama 3.1 405B (Global Batch Size=1024)

Compute:

Roughly 57% of the training iteration wall time (~733 ms) is spent in compute. GEMM kernels dominate, accounting for 74% of that compute time. What is striking is how concentrated the compute is. The FFN up-projection and down-projection GEMMs alone consume ~54% of total compute (~397 ms). Everything else, including QKV projections, attention, softmax, normalization, etc., shares the remaining 46%. the FFN clearly dominates and is the primary driver of compute time.

Communication:

The system operates across two communication domains: high-bandwidth NVIDIA NVLink (NVL) within a rack (scale-up, 72 GPUs) and Azure’s Multipath Reliable Connection (MRC) network across racks (scale-out). The total communication work is 853 GPU-ms. Nearly 42% of this overlaps with compute or other communication (with MRC overlapping NVL), leaving only ~500 ms on the critical path. This is the portion that directly impacts step time, as shown in Figure 1. Of this ~500 ms critical-path communication, NVLink accounts for nearly all of it, while MRC contributes just 1.6%.

Intra-rack NVLink communication on the critical path accounts for 37.8% (~479 ms). This includes Tensor Parallelism (TP), Pipeline Parallelism (PP), and Context Parallelism (CP) exchanges on the 1.8 TB/s intra-rack NVLink fabric, as well as pipeline bubbles, where stages idle between micro-batches. Although substantial, in a weak scaling regime, where each rack handles a fixed shard of the workload, NVLink communication per rack remains constant as we scale.

MRC communication covers gradient synchronization across all DP ranks. Although it represents a substantial amount of communication work, it overlaps heavily with both NVLink communication and compute. As a result, only 1.6% (~20 ms) remains on the critical path. The cross-rack MRC network is nearly invisible in the step profile.

The workload is compute-bound even at 8192 GPU scale, with FFN as the center of gravity. Any improvement to FFN kernel efficiency (precision, tiling, scheduling) translates almost directly into faster training steps.

Why Is Topology-Aware Mapping the Key Enabler of Scale?

In the first section, we showed that cross-rack MRC communication accounts for just 1.6% of step time. This result comes from how parallelism is mapped to the cluster’s two-tier network.

Training Llama 3.1 405B distributes work across four dimensions. Tensor Parallelism (TP), Pipeline Parallelism (PP), and Context Parallelism (CP) split the model and sequence across GPUs, communicating frequently throughout each step. Data Parallelism (DP) replicates the model and synchronizes gradients once per step. The key difference: TP, PP, and CP are latency-sensitive. DP communication can be overlapped with compute, along with other NVLink-based communication. The design principle follows directly: keep latency-sensitive dimensions on the fastest links, and push traffic that can be overlapped with compute to the scale-out fabric

On Azure's Fairwater cluster, each rack provides a 72-GPU NVLink domain at 1.8 TB/s. We use 64 GPUs per rack and constrain TP × PP × CP = 64, so one complete model-parallel shard fits within a single rack. Scaling to more racks adds DP replicas, with gradient synchronization flowing across the 100 GB/s Multipath Reliable Connection (MRC) network, the traffic that hides almost entirely behind compute.

We swept TP, PP, and CP at fixed DP = 8 and GBS = 1152 to isolate the effect of communication efficiency on throughput. TP4–PP8–CP2 achieves the highest TFLOPS/GPU among the feasible configurations, reaching an Model FLOPs Utilization (MFU) of 26.45% as shown in Figure 2.

Figure 2: Avg TFLOPS/GPU for TP/PP/CP configurations at fixed DP=8 and GBS=1152; TP4-PP8-CP2 is the best feasible large-scale choice.

What Limits Scaling Efficiency at 8K+ GPU Scale?

We scaled Llama 3.1 405B training from 40 to 128 racks, adjusting MiniBS (per-DP-shard batch size, i.e., Global Batch Size ÷ DP) and Global Batch Size (GBS) at each scale point to balance GPU utilization against convergence efficiency. Training time drops from 18.79 minutes to 7.07 minutes, representing 83% scaling efficiency relative to the 40-rack baseline as shown in the table below.

Racks	GPUs	MiniBS (GBS/Racks)	Global Batch Size (GBS)	Time-to-Train
40	2,560	24	960	18.79 min
64	4,096	16	1024	12.86 min
112	7,168	8	896	7.75 min
128	8,192	8	1,024	7.07 min

Notice that MiniBS shrinks as we scale. MiniBS determines how much work each GPU processes per step and is the primary driver of step time. As we scale by increasing DP (one replica per rack), the global batch size grows proportionally unless MiniBS is reduced.

Higher MiniBS improves per-step GPU utilization, but at scale it inflates GBS, and beyond a critical batch size the model requires quadratically more samples to converge. So we trade per-step throughput for faster end-to-end convergence, reducing MiniBS from 144 at 8 racks to 8 at 128 racks.

MiniBS cannot be reduced below 8 in our setup because PP = 8 requires at least one microbatch per pipeline stage to keep the pipeline fully utilized. Since MiniBS cannot drop further, adding more racks only increases GBS, pushing deeper into the convergence penalty. At large scale, the optimal configuration minimizes time-to-train, not per-step throughput. The 83% scaling efficiency gap is not a network or system bottleneck. It is inherent to the benchmark's convergence dynamics.

Conclusion

MLPerf Training v6.0 provides a practical lens into how frontier-scale LLM systems behave under real constraints. Azure’s Llama 3.1 405B submission, which achieved a leading result at 8K+ GPU scale, highlights how performance at this level is shaped not just by infrastructure, but by the interaction of compute, topology, and convergence. Scaling efficiently at this level requires aligning these factors with the workload, not just adding more compute.

Acknowledgement

This work was made possible through strong collaboration across multiple teams. We thank Prabhat Ram for his guidance throughout this effort, along with Mark Gitau, Amirreza Rastegari, Ojasvi Bhalerao, Sai Kovouri, Adam Hough, Bhupender Thakur, Nandini Ramanathan, Manasa Govindu, Sanian Gaffar, Ekrem Aksoy, Matthew Kappel, John Rankin, Girish Bhatia, Sreevatsa Anantharamu, Scott Moe, Yang Wang, and Jithin Jose — as well as many others across Azure and NVIDIA who supported this effort.

Scaling High-Performance CAD on Azure Virtual Desktop with NVIDIA RTX PRO 6000

Sunita_AZ0708 — Wed, 17 Jun 2026 22:28:14 GMT

Overview of Siemens NX Multi-Session on Azure

This validation study evaluates how Siemens NX 2506 performs in a multi-user AVD environment, using an RTX-based Azure VM.

The focus was to test:

15 and 30 concurrent users
Single-host vs multi-host configurations
Consistent workloads under identical system configurations

The objective was to determine whether a single RTX PRO 6000 GPU can reliably support 30 users with:

Stable graphics rendering
Consistent user experience
Scalable performance

Azure AVD Architecture for Multi-User NX

The architecture leverages Azure Virtual Desktop with GPU-backed session hosts:

AVD clients connect through Azure control plane (Broker, Gateway)
Session hosts run on Standard NC128ds_v6 RTX PRO 6000 VM
GPU is shared across sessions via vGPU time-slicing
Centralized Siemens License Server ensures consistent access

This setup enables up to 30 concurrent NX sessions on a single host, providing a scalable and centralized engineering environment.Test Configuration (30-User Multisession Focus)

Key setup:

Azure VM: Standard NC128ds_v6
Azure Virtual Desktop : Session Host, Application group, User
GPU: NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM)
CPU/RAM: 128 vCPUs, 512 GB memory
Concurrency: 15 and 30 users
Execution Model: Automated ATS benchmarks
Python

Workloads were executed using the Siemens NX ATS certification suite, covering real-world operations such as:

Display modes, HD3D, raster rendering
Face analysis, multiple views
Visualization performance
BackgroundSettingWireframe, DeEmphasis, DisplaySection, DisplayMode, ExportImage, FaceAnalysis ,GridDisplay,
HD3D, LightDirection, ListingWindow, MirrorDisplay , MultipleViews, RasterImage ,SynchronizeViews, VisualizationPerformance.
These tests covers Real-world Engineering models & Assemblies.

Workload Execution and Validation

All tests were executed in a production-like setup with default VSync, ensuring realistic user behavior.

Automated Python-based execution

Multiple concurrent NX sessions
GPU resource sharing via Azure vGPU

All ATS scenarios completed successfully with PASS results, confirming:

Stable rendering
No crashes or session failures
Reliable GPU sharing

Performance Results: 30 Users on RTX

	Concurrent Users - Single Host		Concurrent Users - Two Hosts		AVD multi Session Host gain(%)
Test Cases	15Users 1VM	30Users 1VM	15Users 2VM	30Users 2VM	15 1VM Vs 15 2VM	30 1VM Vs 30 2VM
BackgroundSettingWireframe	29.67	41.10	30.10	48.96	-1%	-19%
DeEmphasis	67.78	77.21	68.22	72.37	-1%	6%
DisplaySection	55.22	69.24	56.11	63.43	-2%	8%
DisplayMode	35.98	44.33	36.01	36.31	0%	18%
ExportImage	365.72	700.23	246.24	375.08	33%	46%
FaceAnalysis	257.17	280.09	254.32	255.43	1%	9%
GridDisplay	30.91	33.07	26.72	28.27	14%	15%
HD3D	21.76	23.74	19.77	20.73	9%	13%
LightDirection	36.56	37.53	26.57	31.03	27%	17%
ListingWindow	22.88	22.50	16.26	15.35	29%	32%
MirrorDisplay	71.84	74.56	63.16	64.78	12%	13%
MultipleViews	33.25	44.09	32.86	37.04	1%	16%
RasterImage	24.62	32.06	24.15	25.17	2%	21%
SynchronizeViews	25.94	33.43	24.71	25.73	5%	23%
VisualizationPerformance	846.54	1033.23	712.78	826.84	16%	20%

Key Observations (30 Users)

AVD scale-out (multi-session GPU across multiple hosts) reduces resource contention and delivers measurable performance gains, especially at higher user densities.

Single VM (1 GPU) successfully supports 30 users with consistent execution across all test cases
Performance scales predictably as concurrency increases
GPU sharing remains stable even under heavy load

Multi-Host Scaling Advantage

At 30-user scale:

Running across 2 VMs (distributed architecture) improves performance across most workloads
Gains observed in GPU-intensive tasks such as:
- Export Image (~46% improvement)
- Listing Window (~32%)
- Raster Image (~21%)
- Visualization Performance (~20%)

Performance Trends

At 15 users, performance differences between single and dual VM setups are minimal
At 30 users, workload distribution significantly improves responsiveness
Indicates GPU contention becomes more prominent at higher densities

Key Findings

Strong Multi-Session Density on a Single GPU

Successfully validated 30 concurrent NX users on one RTX PRO 6000 GPU
Stable performance across all ATS workloads

Consistent User Experience

Smooth graphics rendering with default VSync
No instability or performance degradation

Scalable Architecture

Single-host deployments are efficient for consolidation
Multi-host deployments improve performance at higher user counts

Enterprise-Ready GPU Virtualization

Azure vGPU sharing enables efficient multi-user utilization
Reliable performance across real-world engineering tasks

Conclusion

This validation confirms that Azure Virtual Desktop with NVIDIA RTX PRO 6000 Blackwell GPUs is a highly effective platform for running Siemens NX 2506 in multi-user environments.

Key takeaways:

✅ A single RTX PRO 6000 GPU can support up to 30 concurrent NX users with stable performance
✅ All certification workloads passed without issues, validating system reliability
✅ Performance remains consistent even under high user density
✅ Scaling across multiple VMs further improves responsiveness at higher loads

This makes Azure AVD a strong candidate for enterprise CAD deployments, enabling:

Centralized GPU infrastructure
Higher user density
Reduced hardware dependency
Scalable engineering workflows

Azure Sets a New Performance Record for LLM Training Benchmark at Extreme Scale

azinheidarshenas — Wed, 17 Jun 2026 00:17:30 GMT

Azure achieved the most performant MLPerf Training v6.0 result to date for Llama 3.1 405B, with a time-to-train of just over seven minutes according to MLCommons. This loadbearing benchmark measures how communication overhead and system stability dominate training performance, where Azure’s full stack, end-to-end advantages shine. Azure scaled to 2,048 NVIDIA GB20 0 NVL72 compute tray nodes spanning 128 racks (8,192 GPUs), assembling the largest reported GB200 NVL72 cluster to date in MLPerf Training. As model sizes continue to grow into hundreds of billions of parameters, achieving this level of performance consistently requires not only massive compute capacity, but also a highly efficient communication infrastructure.

To meet the demands of the massive training workloads, Azure’s Fairwater AI supercomputing infrastructure offers high-performance GPU scale-up domains with resilient, scale-out networking. Fairwater is optimized for frontier-scale distributed AI training, where communication overhead and synchronization latency can become major scaling bottlenecks at multi-thousand GPU scale. This remarkable result was possible by leveraging fifth-generation NVIDIA NVLink at 1,800 GB/s per GPU for intra-rack communication and Azure’s 100 GB/s MRC (Multipath Reliable Connection) fabric accelerated by NVIDIA DOCA and connected with NVIDIA ConnectX-8 and NVIDIA Spectrum-X Ethernet switches for inter-rack communication.

What Enabled Scaling Llama 405B to 8,192 GPUs on Azure

Achieving efficient training at this scale requires more than raw compute. Three architectural ingredients came together to make this result possible:

High operational efficiency of NVLink scale-up domains, providing fast intra-rack GPU-to-GPU communication.
Resiliency and stability of Azure’s MRC scale-out networking fabric, built on NVIDIA ConnectX-8 SuperNICs and Spectrum-X Ethernet switches, delivering 100 GB/s per GPU across racks.
Topology-aware workload mapping, aligning parallelism strategies with the underlying network structure.

Large-scale LLM training is fundamentally synchronous. At every training step, GPUs perform forward and backward passes followed by gradient synchronization across ranks. Because all GPUs must stay in sync, training progress is ultimately limited by the slowest rank — any time spent waiting on the network directly increases overall step time.

For the Llama 405B benchmark, the workload is distributed across four dimensions of parallelism: Tensor, Context, Pipeline, and Data parallelism. Each generates different communication patterns, with some communication sitting directly on the critical training path. If left unmanaged, communication overhead can quickly become a major bottleneck at scale.

The key insight is that not all communication is equally latency-sensitive. Some communication must be completed before computation can continue, while other communication can overlap with compute.

Tensor-Parallel and Context-Parallel communication sit directly on the critical compute path, meaning each layer must wait for its collective operation to complete before execution can proceed. Pipeline-Parallel communication is also on the critical path, but across stages rather than within a layer (i.e., the next stage cannot begin processing its data until the previous stage has transferred its gradients). To minimize latency, all three are placed on the high-bandwidth NVLink domain, which provides up to 1,800 GB/s per GPU. Data-Parallel communication, on the other hand, can overlap with backward compute because gradient reduction runs alongside the backward pass and does not immediately block the next operation. This traffic is carried over the MRC scale-out network.

The impact of such mapping is clear at scale. Our profiling shows that cross-rack MRC communication contributes just ~20 ms of exposed time on the critical path of a ~1.27 s training step (≈1.6% of step time). This communication — primarily gradient synchronization across all ranks — is well overlapped by compute and NVLink traffic, making the scale-out network nearly invisible to training performance.

Stable Step Time as the Cluster Grows

To evaluate execution stability as the cluster scales, we compared our GB200 NVL72 128-rack submission against a 112-rack configuration used for this experiment. In the Llama 405B workload, the critical path of each training step is dominated by compute (forward and backward GEMMs) together with latency-sensitive Tensor-Parallel and Context-Parallel communication over NVLink. Importantly, neither of these changes as the cluster scales out.

Cross-node communication, such as Data-Parallel collectives and Pipeline-Parallel transfers over the MRC network, runs concurrently with backward compute and remains largely hidden behind computation. This overlap is sustained only if cross-node communication remains fast and predictable. Any network instability, congestion, or synchronization jitter would reduce this overlap and expose communication on the critical path, directly increasing overall step time.

Figure 1. Step-time vs cluster size at scale

This is exactly what we observe in practice under a weak scaling model, where we maintain a constant workload per GPU while scaling total cluster capacity. As shown in Figure 1, step time remains nearly identical at scale, measuring 1.2734 seconds at 112 GB200 NVL72 racks (7,168 GPUs) and 1.2712 seconds at 128 racks (8,192 GPUs) — a difference of just 2 ms. This corresponds to a near-perfect 99.8% weak scaling efficiency as we expanded the cluster by an additional 1,024 GPUs. Step-time variance also remains extremely low (±0.04–0.05%), confirming stable and predictable execution at extreme scale. This directly translates to maximized hardware utilization and an accelerated overall time-to-train, demonstrating the resiliency of Azure’s MRC scale-out network.

Acknowledgement

This work was made possible through strong collaboration across multiple teams. We would like to especially recognize Shantanu Patankar for his contributions to execution and performance analysis and Mark Gitau for his contributions to data preparation and experiment support as core contributors, and extend our thanks to Amirreza Rastegari, Ojasvi Bhalerao, Sai Kovouri, Adam Hough, Nandini Ramanathan, Manasa Govindu, Sanian Gaffar, Ekrem Aksoy, Bhupender Thakur, Matthew Kappel, John Rankin, Girish Bhatia, Sreevatsa Anantharamu, Scott Moe, Yang Wang, and Jithin Jose, as well as many others across Azure and NVIDIA who supported this effort.

Real-Time Simulation with Ansys Discovery on Azure AMD Radeon V710 GPUs

Sunita_AZ0708 — Thu, 11 Jun 2026 14:09:17 GMT

Overview of Synopsys-Ansys Discovery

Ansys Discovery is an interactive engineering application that unifies 3D geometry modeling, real‑time simulation, and visual analysis into a single, immersive workspace. It is purpose‑built to accelerate early‑stage engineering exploration while maintaining continuity into high‑fidelity validation workflows.

Discovery enables engineers and designers to create, modify, simulate, and iterate on designs in real time, dramatically reducing the time required to assess design feasibility, physics behavior, and performance tradeoffs. Unlike traditional sequential CAD‑then‑simulation workflows, Discovery collapses these steps into a continuous feedback loop that supports faster decision‑making and more informed design outcomes.

Ansys Discovery is organized into three continuously accessible stages:

Model – Focused on direct 3D geometry creation and modification, without simulation overhead
Explore – Provides instant, real‑time simulation results using GPU‑accelerated meshing and solvers for rapid insight
Refine – Enables higher‑fidelity simulation using body‑fitted meshing and CPU/GPU solvers for design validation

Running Discovery on Azure extends these capabilities by enabling scalable, cloud-based engineering workflows.

Key benefits

Running Ansys Discovery on Azure provides a scalable, cloud-based platform for simulation-driven design with the following advantages:
Access high-performance engineering workstations from anywhere
Azure provides global access to GPU-enabled virtual machines, enabling engineers to run Discovery from any location without dependency on local hardware.
Engineers no longer need high-end local GPUs, as compute and visualization are delivered through Azure-hosted environments. This reduces hardware refresh cycles and IT overhead.
Azure NVads V710 v5 supports fractional GPU allocation (1/6 to full GPU), allowing organizations to right-size cost and performance based on workload requirements.
Discovery’s GPU-accelerated Explore mode benefits from Azure GPU infrastructure, enabling instant physics feedback and faster design iteration cycles.
CAD models, simulation results, and design artifacts can be stored in Azure Files or Azure NetApp Files, reducing data duplication and improving collaboration across teams.
Azure allows rapid provisioning and scaling of GPU resources to match peak design cycles or project demands.
Intellectual property remains within enterprise-managed Azure environments, with access controlled using Microsoft Entra ID and Azure security services.
Cloud-hosted Discovery environments enable geographically distributed teams to collaborate on shared designs with consistent performance and user experience.

To support these capabilities, Azure provides a reference architecture optimized for GPU-accelerated simulation.

Architecture

The architecture for running Ansys Discovery on Azure is designed to deliver interactive, GPU‑accelerated simulation in a scalable and cloud-native environment. As illustrated in the diagram, engineers connect to Azure-hosted virtual workstations powered by NVads V710 v5 GPUs, enabling real-time geometry modeling and simulation through remote visualization. Design data and simulation outputs are stored in shared cloud storage services such as Azure Files or Azure NetApp Files, ensuring consistency and accessibility across distributed teams. This setup allows users to seamlessly move between interactive exploration and higher-fidelity validation workflows, while benefiting from on-demand scaling, centralized security, and reduced dependency on local high-performance hardware.

Azure GPU SKU Mapping

Architectural Mapping Overview

Ansys Discovery workloads on Azure naturally align to two primary GPU VM categories:

Graphics‑optimized, fractional GPU -> Design exploration
Full‑GPU -> Design Refinement

This mapping enables right‑sizing of cost, performance, and user experience on the following recommended Azure VM Series.

Azure NVads V710 v5series
NVads V710 v5 VMs are designed for graphics‑intensive, interactive workloads and provide:
- AMD Radeon™ Pro V710 GPUs
- Fractional GPU support (1⁄6 → full GPU)
- Cost‑optimized GPU visualization
- High‑frame‑rate remote graphics
- No additional GPU licensing required
These characteristics align directly with Discovery’s real‑time, GPU‑accelerated Explore mode, where instant feedback matters more than absolute solver depth.
Typical Discovery Workloads on NVads V710 v5
- Interactive geometry creation and modification
- Real‑time physics feedback in Explore mode
- Concept‑level CFD, structural, and thermal studies
- Design review sessions and rapid iteration loops
Early‑stage topology and shape experimentation

Recommended Sizes (Right‑Sizing Guidance)

Discovery Use Case	Azure VM Size	GPU Allocation
Single engineer, light models	Standard_NV4ads_V710_v5	1⁄6 GPU
Moderate assemblies, daily use	Standard_NV8ads_V710_v5	1⁄3 GPU
Large CAD models, advanced explore	Standard_NV12ads_V710_v5	1⁄2 GPU
Power users, complex scenes	Standard_NV24ads_V710_v5	Full GPU

Test Scenarios:

Test Name	Simulation Analysis Type
External Aerodynamics Test 2	Fluid
Internal Flow Test 1	Fluid
Internal Flow Test 2	Fluid
Internal Flow Test 3	Fluid
Heat Transfer Test 1	Thermal
Conjugate Heat Transfer Test 1	Fluid-Thermal
Conjugate Heat Transfer Test 2	Fluid-Thermal
Static Structural Test 1	Structural
Static Structural Test 2	Structural
Static Structural Assembly Test 1	Structural
Static Structural Assembly Test 2	Structural
Natural Frequency Test 1	Modal
Natural Frequency Test 2	Modal

Result Table:

Test Name	Solve Time (s)	Element Count	Test Result
External Aerodynamics Test 2	47.68	1.44E+06	Pass
Internal Flow Test 1	26.70	7.90E+05	Pass
Internal Flow Test 2	9.34	9.64E+05	Pass
Internal Flow Test 3	18.00	1.39E+06	Pass
Heat Transfer Test 1	4.55	2.51E+06	Pass
Conjugate Heat Transfer Test 1	12.34	1.34E+06	Pass
Conjugate Heat Transfer Test 2	30.89	1.54E+06	Pass
Static Structural Test 1	4.76	3.06E+06	Pass
Static Structural Test 2	4.78	3.04E+06	Pass
Static Structural Assembly Test 1	12.98	2.97E+06	Pass
Static Structural Assembly Test 2	14.16	1.88E+06	Pass
Natural Frequency Test 1	36.04	1.45E+06	Pass
Natural Frequency Test 2	16.10	1.11E+06	Pass

These results demonstrate consistent performance across fluid, thermal, and structural simulations, with sub-minute solve times even for multi-million element models.

Conclusion:

The validation confirms that Azure NVads V710 v5 delivers consistent, scalable performance across diverse simulation workloads. With fractional GPU flexibility and strong visualization capabilities, it enables organizations to adopt real-time simulation without overprovisioning infrastructure.

All workloads successfully passed, demonstrating consistent performance across simulation types. Importantly, we’re seeing rapid solve times even at multi-million element scale, which reinforces the ability to use Discovery for real-time engineering exploration at enterprise level. Azure NVads V710 v5 is uniquely optimized for Discovery’s real-time simulation workflows. It provides GPU acceleration tuned for interactive engineering, not just batch compute.

The fractional GPU model allows organizations to right-size cost and performance based on user needs—from individual engineers to power users. This ensures maximum flexibility while maintaining high-quality visualization and responsiveness.

Training 100B+ Models on a Single GPU: What MegaTrain Changes - and What It Means for Azure

yuvmaz — Fri, 29 May 2026 06:08:50 GMT

The Paradigm Shift in Model Training

The conventional wisdom in deep learning has been simple: bigger models require bigger infrastructure. Training a 100-billion parameter language model traditionally demands massive GPU clusters with terabytes of combined memory, where each GPU holds portions of the model simultaneously. This assumption has shaped the entire AI infrastructure landscape, driving demand for high-memory accelerators and complex distributed training frameworks.

A new paper named MegaTrain (Yuan et al) by research teams at NotreDame and Lehigh universities, changes this paradigm by demonstrating that model size need not be limited by GPU memory capacity. Instead of treating memory as a container that must hold the entire model, MegaTrain treats it as a cache through which model components flow during computation. This architectural inversion enables training models orders of magnitude larger than available GPU memory, transforming what was thought to be a hardware limitation into a software scheduling problem.

The implications extend beyond academic curiosity to practical infrastructure decisions, particularly for cloud platforms like Azure where GPU configurations, interconnect topologies, and cost structures create distinct optimization landscapes.

The Memory Bottleneck in Traditional LLM Training

Large language model training faces a fundamental memory constraint that grows with model scale. A typical 100-billion parameter model with mixed-precision training may require in excess of 1 Terabyte of memory when taking into account weights and optimizer states. Activation memory adds another substantial overhead, particularly for long sequences where intermediate layer outputs must be retained for backpropagation.

Traditional training approaches keep all parameters, gradients, and optimizer states resident in GPU memory simultaneously, creating a hard ceiling on trainable model size. Standard datacenter GPUs like the NVIDIA A100 with 80GB memory can barely accommodate models beyond 20 billion parameters with reasonable batch sizes, while consumer-grade GPUs with 16-24GB memory are restricted to models under 5 billion parameters.

This memory wall has forced the AI community toward distributed training strategies like pipeline parallelism, tensor parallelism, and data parallelism, each adding communication overhead, synchronization complexity, and infrastructure costs. The memory bottleneck becomes particularly acute during gradient accumulation and optimizer updates, where transient memory peaks can trigger out-of-memory errors even when average utilization suggests sufficient capacity.

The Key Insight: Inverting the Memory Hierarchy

MegaTrain's core innovation inverts the traditional memory hierarchy by treating GPU memory as a streaming cache rather than a static container. In conventional training, the model resides in GPU memory while slower storage sits idle except for checkpoint loading. MegaTrain reverses this relationship: the full model lives in high-bandwidth storage like NVMe SSDs, with only the actively computing layer residing in GPU memory at any moment.

This architectural inversion exploits the sequential nature of neural network computation: forward and backward passes process layers in deterministic order, creating predictable access patterns amenable to prefetching and streaming. The approach transforms the memory constraint from a capacity problem into a bandwidth problem where success depends on streaming layers between storage and GPU fast enough to keep compute units saturated.

Modern NVMe SSDs provide 7-14 GB/s sequential read bandwidth, while PCIe 4.0 x16 offers 32 GB/s bidirectional throughput, creating sufficient headroom for layer streaming when properly pipelined. This memory hierarchy is the basis for a fundamental architectural shift, through which MegaTrain moves from memory-bound scaling to bandwidth-bound scaling.

MegaTrain Architecture and Core Mechanisms

MegaTrain implements memory hierarchy inversion through four interlocking mechanisms that together enable efficient single-GPU training of massive models. The architecture maintains model parameters, optimizer states, and gradients in host memory or NVMe storage, streaming only the actively computing layer into GPU memory. Each training iteration processes layers sequentially: during the forward pass, layers stream from storage to GPU, compute activations, then stream back to make room for the next layer. The backward pass reverses this flow, streaming layers in reverse order to compute gradients. Critically, MegaTrain maintains minimal GPU memory footprint by immediately evicting each layer after computation rather than accumulating them.

The system achieves this through careful orchestration of four core techniques that transform naive layer streaming from a theoretical possibility into a practical training method.

Layer-wise streaming provides the foundation by decomposing model computation into independent sequential operations.
Pipelined execution overlaps data movement with computation to hide transfer latency.
Block-wise recomputation trades redundant forward passes for reduced activation memory.
Stateless execution eliminates persistent GPU state to maximize available memory for active computation.

Together, these techniques enable training throughput within 2-5x of conventional in-memory training while supporting models 10-100x larger than GPU memory capacity. Let’s look at each of these techniques in detail.

Layer-Wise Streaming: Sequential Model Decomposition

Layer-wise streaming exploits the inherently sequential structure of neural network computation to decompose model execution into memory-independent stages. Each transformer layer performs a self-contained computation: it receives input activations, applies attention and feedforward transformations, and produces output activations for the next layer. MegaTrain leverages this modularity by loading exactly one layer's parameters into GPU memory, computing its forward pass, storing the output activations, then immediately evicting the layer to make room for the next. During backpropagation, the process reverses: layers stream in reverse order, recompute their forward pass from stored input activations, compute gradients, and stream back to storage.

This approach reduces peak GPU memory from the sum of all layer parameters to the size of the single largest layer plus activation storage. For a 100-billion parameter model with 80 transformer layers, each layer contains approximately 1.25 billion parameters requiring 2.5GB in FP16, compared to 200GB for the full model. The memory savings enable training on GPUs that would otherwise be incapable of holding even a fraction of the model.

Layer-wise streaming does introduce computational overhead: each layer must be loaded and evicted during both forward and backward passes, creating 4x the parameter transfer volume compared to conventional training. However, modern interconnect bandwidth and careful prefetching largely mitigate this overhead when properly pipelined.

Pipelined Execution: Overlapping Transfer and Compute

Pipelined execution hides layer streaming latency by overlapping data transfers with GPU computation, ensuring compute units remain saturated despite continuous parameter movement. MegaTrain maintains a three-stage pipeline: while the GPU computes layer N, the system prefetches layer N+1 from storage to host memory and simultaneously evicts layer N-1 back to storage. This overlapping execution exploits the independence of CPU-GPU data transfers and GPU computation in modern architectures.

PCIe transfers and GPU kernels execute concurrently when properly scheduled using asynchronous CUDA streams, allowing near-complete latency hiding if transfer time does not exceed computation time. For typical transformer layers, forward pass computation takes 50-200ms depending on batch size and sequence length, while transferring a 2.5GB layer over PCIe 4.0 x16 at 16 GB/s requires only 156ms, fitting comfortably within the computation window.

The pipeline achieves optimal efficiency when transfer bandwidth and computation throughput are balanced, creating a predictable performance model based on layer size, batch configuration, and interconnect speed. The three-stage pipeline pattern - prefetch, compute, and eviction - interleaves to maintain continuous GPU utilization, as illustrated in the diagram above.

Block-Wise Recomputation

Block-wise recomputation addresses the activation memory bottleneck by trading computation for memory through selective forward pass recalculation. In standard training, all intermediate activations from the forward pass must be retained for gradient computation during backpropagation, consuming memory proportional to model depth and batch size. MegaTrain instead stores only a subset of checkpoint activations at regular intervals, recomputing intermediate values on-demand during the backward pass. For a model with 80 layers, storing checkpoints every 10 layers reduces activation memory by 90 percent at the cost of recomputing 9 out of 10 layers during backpropagation. This trade-off proves favorable because forward pass computation is relatively cheap compared to the memory savings, and recomputation can be pipelined with layer streaming.

Stateless Execution

Stateless execution complements recomputation by eliminating all persistent GPU state between layer computations. After processing each layer, MegaTrain immediately evicts not only parameters but also optimizer states like momentum buffers and variance estimates, keeping only the minimal activations needed for gradient flow. This aggressive state eviction maximizes available GPU memory for the active layer's computation, enabling larger batch sizes and longer sequences within fixed memory budgets. Together, block-wise recomputation and stateless execution transform MegaTrain's memory footprint from O(model_size) to O(largest_layer + checkpoint_activations), typically reducing requirements by 50-100x for models in the 100-billion parameter range.

W- Weight prefetch, F/R/B – Computation, G – Gradient offload

Source: Yuan, Zhengqing, et al. MegaTrain: Full Precision Training of 100B+ Parameter Large Language Models on a Single GPU

Performance Results and Achievements

MegaTrain demonstrates practical viability by training models far exceeding GPU memory capacity with acceptable performance overhead. The paper reports an implementation that successfully trained a 175-billion parameter model on a single NVIDIA A100 GPU with 80GB memory. Using conventional techniques, that same hardware configuration would support only 15-20 billion parameters. When using this conventional configuration as the baseline to compare to, MegaTrain throughput reached 45 percent of the baseline in-memory training speed, translating to approximately 2.2x wallclock time for equivalent training steps. For smaller models at 30-billion parameters, throughput exceeded 65 percent of baseline as the ratio of computation to transfer improved.

The chart demonstrates the inverse relationship between model size and throughput efficiency. As model parameters increase from 30 billion to 175 billion, throughput degrades linearly from 65% to 45% of baseline, validating the predictable performance characteristics of the streaming approach. This degradation reflects the increasing ratio of layer count to computation time as models grow larger.

Memory efficiency gains are substantial: peak GPU memory utilization remained under 40GB throughout training regardless of model size, with the remainder allocated to batch size expansion that partially offset streaming overhead. The system sustained 12-14 GB/s effective storage bandwidth during training, approaching theoretical NVMe limits and validating the pipelined streaming approach.

The memory utilization chart reveals MegaTrain's fundamental breakthrough: GPU memory consumption remains nearly constant across vastly different model sizes. While conventional training would require proportionally more GPU memory as models grow, MegaTrain keeps peak utilization between 35GB and 40GB regardless of whether the model has 30 billion or 175 billion parameters. This flat memory profile enables single-GPU training of models that would otherwise require expensive multi-GPU configurations.

Scaling analysis revealed predictable performance characteristics: throughput degraded linearly with model size as layer count increased, while improving with batch size as computation became more dominant relative to fixed transfer costs. Energy efficiency comparisons showed single-GPU MegaTrain consuming 85 percent less power than equivalent 8-GPU distributed training for the same model, though at 2.2x longer duration resulting in 40 percent total energy savings.

Running MegaTrain on Azure NC-Series GPUs

Azure's NC-series GPU virtual machines provide diverse configurations for MegaTrain deployment, each with distinct performance characteristics shaped by GPU type, interconnect topology, and storage options. The NC A100 v4 series offers NVIDIA A100 GPUs with 80GB memory over PCIe 4.0 interfaces, directly matching the original MegaTrain research configuration. These instances provide 32 GB/s bidirectional PCIe bandwidth sufficient for layer streaming without bottlenecking on parameter transfers for models up to 200-billion parameters. Storage configuration critically impacts MegaTrain performance: Azure Premium SSD v2 delivers up to 16 GB/s throughput with appropriate provisioning, while local NVMe SSDs on storage-optimized instances can reach 7-14 GB/s depending on VM size. The NC series of VMs provides a large amount of system memory, local NVMe storage and GPUs, creating an ideal environment for MegaTrain's host memory and storage requirements. Importantly, Azure's NC-series instances use PCIe-connected GPUs rather than NVLink-connected configurations, making interconnect bandwidth the primary performance constraint rather than GPU compute capacity. Cost analysis favors MegaTrain for experimentation and medium-scale training: a single NC A100 v4 instance costs ~4 dollars per hour compared ~30 dollars per hour for an 8-GPU NDv4 instance, enabling 8x cost reduction for workloads tolerant of 2-3x longer training duration.

Code for MegaTrain is available on the project’s Github repository at https://github.com/DLYuanGod/MegaTrain. Supported models at the time of writing include Qwen 2/2.5/3/3.5/3.5 MoE, Llama 2/3/3.1/3.2/3.3/4, Mistral, Mixtral, DeepSeek Code/R1, Phi-3/4, Gemma 2/3, as well as some others.

Critical Constraint: PCIe Bandwidth vs NVLink

The interconnect bandwidth hierarchy creates fundamentally different performance envelopes for MegaTrain on Azure compared to specialized research hardware. PCIe 4.0 x16 connections on Azure NC-series instances provide 32 GB/s bidirectional bandwidth between CPU and GPU.

For MegaTrain's layer streaming workload, CPU-GPU bandwidth matters most since model parameters flow from host memory or NVMe storage through PCIe to GPU memory. PCIe 4.0's 32 GB/s bidirectional capacity proves sufficient for models up to 200-billion parameters when combined with aggressive prefetching and eviction pipelining but becomes a hard bottleneck for larger models where layer transfer time exceeds computation time.

Trade-Offs and Limitations

MegaTrain introduces several trade-offs that practitioners must evaluate against infrastructure constraints and training objectives. Training throughput degrades to 40-65 percent of baseline depending on model size, extending wallclock training time by 1.5-2.5x for equivalent iteration counts. This slowdown proves acceptable for research experimentation and low-frequency retraining but may be prohibitive for production pipelines requiring rapid iteration.

Memory efficiency gains come at the cost of increased storage I/O: a full training run generates 4x the parameter transfer volume compared to conventional training due to bidirectional streaming during forward and backward passes. This I/O amplification accelerates SSD wear and may impact cost for cloud storage with throughput-based pricing.

Implications for AI Infrastructure and Azure Strategy

MegaTrain's viability reshapes infrastructure planning assumptions for AI training workloads, particularly in cost-constrained environments where model size exceeds available GPU memory. Organizations can defer expensive hardware upgrades by exploiting existing GPU capacity more fully through memory hierarchy inversion, extending the useful life of current hardware generations. Cloud platforms like Azure benefit from increased flexibility in instance sizing: users can select GPU types based on compute requirements rather than being forced into high-memory configurations solely for capacity. This decoupling enables better price-performance optimization by matching compute intensity to GPU type while relying on storage for capacity scaling. The approach particularly suits Azure's NC-series positioning as a cost-effective alternative to specialized AI instances, turning PCIe bandwidth into an acceptable constraint rather than a disqualifying limitation. Research teams gain the ability to prototype and validate large model architectures without provisioning expensive multi-GPU clusters, accelerating iteration during early development phases.

However, production deployment strategies should carefully evaluate MegaTrain's throughput trade-offs against distributed training alternatives: the 2-3x slowdown may be unacceptable for latency-sensitive pipelines despite cost savings. Infrastructure teams should prioritize NVMe storage provisioning and PCIe 4.0/5.0 support (for example, by using newer GPU types such as the H100) when deploying MegaTrain-compatible environments, as storage bandwidth directly determines achievable performance. Long-term, the techniques validate a broader trend toward memory-disaggregated computing where storage, memory, and compute scale independently rather than in fixed ratios determined by hardware packages.

AI Infrastructure Preflight at User space: Validating Multi Node, Multi GPU Slurm Clusters

vinilv — Fri, 22 May 2026 12:29:15 GMT

Every team that operates GPU clusters for AI has seen this pattern. The cluster boots, GPUs are visible, and scheduling works at a basic level. Then the first distributed training run stalls in NCCL initialization, fails during rank rendezvous, or silently maps ranks to the wrong devices. The issue is often not in training code. It is in infrastructure consistency across scheduler, runtime, drivers, networking, and process topology.

The goal of ai-infra-validator is straightforward:

Run a fast user space preflight before expensive training jobs.
Validate distributed initialization for multi node, multi GPU workloads.
Confirm GPU affinity and rank mapping are correct.
Verify NCCL communication fabric can complete a collective ring under Slurm.

This post walks through the implementation in detail, explains why each part exists, and shows how to operationalize it in real HPC AI environments.

What the project validates

Zero-dependency user space smoke test for AI clusters. Validates multi-node PyTorch DDP initialization, GPU affinity, and NCCL fabric connectivity under Slurm orchestration.

Git Repo: ai-cluster-validator

In practical terms, this checks that:

Slurm launches the expected number of ranks per node.
Distributed process group creation with NCCL succeeds.
Each rank binds to the expected local GPU.
Cross-rank all-reduce completes and converges.
Node level telemetry confirms software and fabric state.

This is not a performance benchmark. It is a correctness and readiness gate.

Tested platform profile

Component	Value
CycleCloud	8.8.3-3667
Slurm	25.05.5
Slurm partition	hpc
Scheduler VM SKU	Standard_D8s_v6
Compute VM SKU	Standard_ND96asr_v4
OS images	microsoft-dsvm:ubuntu-hpc:2204:latest and microsoft-dsvm:ubuntu-hpc:2404:latest
PyTorch	2.12.0+cu130
CUDA runtime	13.0
NCCL target	2.29.7

This profile represents a common enterprise scenario where scheduler and compute nodes have different roles, and the training fleet depends on correct multi node orchestration.

Step 1: Minimal user space bootstrap

The bootstrap script creates a shared Python environment at /shared/apps/pytorch_env and installs the required packages:

torch
torchvision
torchaudio
psutil

This choice is intentional:

No dependency on containers for first-pass validation.
Single environment path visible to all compute nodes.
Rapid setup and repeatability for cluster operators.

Command sequence:

git clone https://github.com/vinil-v/ai-cluster-validator.git cd ai-cluster-validator sudo bash bootstrap_env.sh

Step 2: Slurm job defines deterministic distributed topology

The Slurm script expresses a clear topology contract:

nodes=2
ntasks-per-node=8
gpus-per-node=8
cpus-per-task=12

From this, world size is derived as:

WORLD_SIZE = SLURM_NTASKS = 2 x 8 = 16

The script also configures network and NCCL behavior:

NCCL_DEBUG=WARN
NCCL_IB_DISABLE=0
NCCL_P2P_DISABLE=0
NCCL_IGNORE_CPU_AFFINITY=1
GLOO_SOCKET_IFNAME=eth0
NCCL_SOCKET_IFNAME=eth0

Important implementation detail:

MASTER_ADDR is set to the first host in SLURM_JOB_NODELIST.
MASTER_PORT is selected dynamically from the ephemeral range 49152-65535 and falls back to 29500 if needed.

Why this matters:

Reduces port collision risk when jobs run frequently.
Avoids hardcoded rendezvous values that may fail in shared clusters.

Launch path:

srun --cpu-bind=none bash -c " source /shared/apps/pytorch_env/bin/activate; export RANK=$SLURM_PROCID; export LOCAL_RANK=$SLURM_LOCALID; python3 ddp_mesh_ping.py "

The LOCAL_RANK handoff is critical for stable GPU affinity inside each node.

Step 3: DDP initialization and rank to GPU affinity

Inside ddp_mesh_ping.py, each process executes:

Parse WORLD_SIZE, RANK, LOCAL_RANK, MASTER_ADDR, MASTER_PORT.
Initialize torch.distributed with backend nccl and TCP init method.
Set CUDA device using LOCAL_RANK.

Core initialization path:

dist.init_process_group( backend="nccl", init_method=f"tcp://{master_addr}:{master_port}", world_size=world_size, rank=rank ) torch.cuda.set_device(local_rank)

This validates the minimum distributed contract required by real model training jobs.

Step 4: Rich node and fabric telemetry in user space

Each rank collects detailed metadata before the collective test:

Node identity from Slurm and hostname.
GPU model and VRAM from CUDA properties.
System memory via psutil.
CPU model from /proc/cpuinfo.
OS and kernel versions.
NVIDIA driver version from /proc/driver/nvidia/version.
PyTorch, CUDA, and NCCL runtime versions.
InfiniBand device state and link rate from /sys/class/infiniband.
Basic GPU peer access capability via torch.cuda.can_device_access_peer.

All rank payloads are gathered on rank 0 using dist.gather_object and printed as:

Cluster hardware topology report.
Node environment deep dive.
Network interconnect and fabric status.

This design gives platform teams one artifact that is both operational and diagnostic.

Step 5: Functional collective validation

After telemetry, each rank executes a lightweight DDP compute path:

Build nn.Linear(10,10) on local GPU.
Wrap with DistributedDataParallel.
Perform forward, loss, backward.
Run all_reduce on loss tensor.
Compute global average loss.

Pass condition is explicit in log output:

SUCCESS: DDP Multi-Node AllReduce Ring Complete!

This confirms that process group initialization and collective communication both completed successfully.

What a successful run looks like

Submission:

sbatch ddp_smoke_test.slurm squeue

Representative outcomes in the log:

Total Execution Ranks: 16
Two nodes with local ranks 0 through 7 on each node
GPU inventory aligned with expected A100 topology
Active InfiniBand HCAs discovered per host
NCCL socket interface set to eth0
Final success marker and computed convergence loss

When these markers are present and coherent with expected hardware shape, the cluster is typically ready for distributed training bring-up.

How to check the output file

The Slurm script writes two artifacts per job:

ai_infra_smoke_test_<jobid>.log
ai_infra_smoke_test_<jobid>.err

Use this exact workflow after submission:

# 1. Submit and capture the job id sbatch ddp_smoke_test.slurm # 2. Check job state squeue -j <jobid> # 3. Read standard output log cat ai_infra_smoke_test_<jobid>.log # 4. Read standard error log cat ai_infra_smoke_test_<jobid>.err

For stronger validation in automation, also check:

Total Execution Ranks equals expected world size.
Both nodes appear in the topology table with local ranks 0 through 7.
NCCL/CUDA/PyTorch versions are present in the node environment section.

Complete reference output

Use the following full log as a known-good reference from a successful 2-node ND96asr_v4 run.

Master Node IP/Hostname: ddpcluster-hpc-1 Dynamically Assigned Port: 53593 Total Execution Ranks: 16 =============================================================================================== HPC CLUSTER INTERACTION MONITOR =============================================================================================== --> Initializing DDP on Master Node : ddpcluster-hpc-1 --> Dynamic Coordination Port : 53593 --> Target World Cluster Size : 16 GPUs ----------------------------------------------------------------------------------------------- =============================================================================================== CLUSTER HARDWARE TOPOLOGY REPORT =============================================================================================== | Rank | Node Name | Local ID | GPU Model | VRAM | Sys Mem | CPU Cores | ----------------------------------------------------------------------------------------------- | 0 | ddpcluster-hpc-1 | 0 | NVIDIA A100-SXM4-4 | 39.5 GB | 885.8 GB | 96 Cores | | 1 | ddpcluster-hpc-1 | 1 | NVIDIA A100-SXM4-4 | 39.5 GB | 885.8 GB | 96 Cores | | 2 | ddpcluster-hpc-1 | 2 | NVIDIA A100-SXM4-4 | 39.5 GB | 885.8 GB | 96 Cores | | 3 | ddpcluster-hpc-1 | 3 | NVIDIA A100-SXM4-4 | 39.5 GB | 885.8 GB | 96 Cores | | 4 | ddpcluster-hpc-1 | 4 | NVIDIA A100-SXM4-4 | 39.5 GB | 885.8 GB | 96 Cores | | 5 | ddpcluster-hpc-1 | 5 | NVIDIA A100-SXM4-4 | 39.5 GB | 885.8 GB | 96 Cores | | 6 | ddpcluster-hpc-1 | 6 | NVIDIA A100-SXM4-4 | 39.5 GB | 885.8 GB | 96 Cores | | 7 | ddpcluster-hpc-1 | 7 | NVIDIA A100-SXM4-4 | 39.5 GB | 885.8 GB | 96 Cores | | 8 | ddpcluster-hpc-2 | 0 | NVIDIA A100-SXM4-4 | 39.5 GB | 885.8 GB | 96 Cores | | 9 | ddpcluster-hpc-2 | 1 | NVIDIA A100-SXM4-4 | 39.5 GB | 885.8 GB | 96 Cores | | 10 | ddpcluster-hpc-2 | 2 | NVIDIA A100-SXM4-4 | 39.5 GB | 885.8 GB | 96 Cores | | 11 | ddpcluster-hpc-2 | 3 | NVIDIA A100-SXM4-4 | 39.5 GB | 885.8 GB | 96 Cores | | 12 | ddpcluster-hpc-2 | 4 | NVIDIA A100-SXM4-4 | 39.5 GB | 885.8 GB | 96 Cores | | 13 | ddpcluster-hpc-2 | 5 | NVIDIA A100-SXM4-4 | 39.5 GB | 885.8 GB | 96 Cores | | 14 | ddpcluster-hpc-2 | 6 | NVIDIA A100-SXM4-4 | 39.5 GB | 885.8 GB | 96 Cores | | 15 | ddpcluster-hpc-2 | 7 | NVIDIA A100-SXM4-4 | 39.5 GB | 885.8 GB | 96 Cores | =============================================================================================== NODE ENVIRONMENT DEEP DIVE ----------------------------------------------------------------------------------------------- [ddpcluster-hpc-1] Details: --> CPU Microarchitecture : AMD EPYC 7V12 64-Core Processor --> Operating System : Ubuntu 22.04.5 LTS --> Kernel Base Version : 5.15.0-1110-azure --> Nvidia Driver Loaded : 580.126.20 --> PyTorch Environment : v2.12.0+cu130 --> CUDA Runtime Version : v13.0 --> NCCL Fabric Target : v2.29.7 --> Discovered InfiniBand HCAs: - mlx5_an0:1 (4: ACTIVE - 40 Gb/sec (4X QDR)) - mlx5_ib0:1 (4: ACTIVE - 200 Gb/sec (4X HDR)) - mlx5_ib1:1 (4: ACTIVE - 200 Gb/sec (4X HDR)) - mlx5_ib2:1 (4: ACTIVE - 200 Gb/sec (4X HDR)) - mlx5_ib3:1 (4: ACTIVE - 200 Gb/sec (4X HDR)) - mlx5_ib4:1 (4: ACTIVE - 200 Gb/sec (4X HDR)) - mlx5_ib5:1 (4: ACTIVE - 200 Gb/sec (4X HDR)) - mlx5_ib6:1 (4: ACTIVE - 200 Gb/sec (4X HDR)) - mlx5_ib7:1 (4: ACTIVE - 200 Gb/sec (4X HDR)) ----------------------------------------------------------------------------------------------- [ddpcluster-hpc-2] Details: --> CPU Microarchitecture : AMD EPYC 7V12 64-Core Processor --> Operating System : Ubuntu 22.04.5 LTS --> Kernel Base Version : 5.15.0-1110-azure --> Nvidia Driver Loaded : 580.126.20 --> PyTorch Environment : v2.12.0+cu130 --> CUDA Runtime Version : v13.0 --> NCCL Fabric Target : v2.29.7 --> Discovered InfiniBand HCAs: - mlx5_an0:1 (4: ACTIVE - 40 Gb/sec (4X QDR)) - mlx5_ib0:1 (4: ACTIVE - 200 Gb/sec (4X HDR)) - mlx5_ib1:1 (4: ACTIVE - 200 Gb/sec (4X HDR)) - mlx5_ib2:1 (4: ACTIVE - 200 Gb/sec (4X HDR)) - mlx5_ib3:1 (4: ACTIVE - 200 Gb/sec (4X HDR)) - mlx5_ib4:1 (4: ACTIVE - 200 Gb/sec (4X HDR)) - mlx5_ib5:1 (4: ACTIVE - 200 Gb/sec (4X HDR)) - mlx5_ib6:1 (4: ACTIVE - 200 Gb/sec (4X HDR)) - mlx5_ib7:1 (4: ACTIVE - 200 Gb/sec (4X HDR)) ----------------------------------------------------------------------------------------------- NETWORK INTERCONNECT & FABRIC STATUS ----------------------------------------------------------------------------------------------- --> Target Communication Interface (NCCL_SOCKET_IFNAME) : eth0 --> Active Telemetry Tracking Level (NCCL_DEBUG) : WARN --> Inter-GPU Topo Link Verification : Active (P2P/NVLink Capable) ----------------------------------------------------------------------------------------------- SUCCESS: DDP Multi-Node AllReduce Ring Complete! --> Computed System Verification Convergence Loss : 1.398719 ===============================================================================================

Why this is effective for platform operations

For AI infrastructure teams, this pattern is highly effective because it is:

Fast: can be run after every change window.
Deterministic: same topology contracts every run.
Actionable: output includes enough context for first-level triage.
Low friction: user space only, no heavy control plane dependencies.

This supports common operating workflows:

Day-0 cluster acceptance.
Day-1 patch validation after driver, kernel, or image changes.
Regression gate in golden image pipelines.
Preflight before large multi node model training jobs.

Practical guidance for extending to larger clusters

Adjust Slurm directives for nodes and tasks per node.
Keep one rank per GPU unless validating alternate placement policy.
Set NCCL_SOCKET_IFNAME and GLOO_SOCKET_IFNAME according to your network policy.
Preserve the dynamic MASTER_PORT logic to avoid static collisions.
Keep the success marker string stable so automation can parse it.

Closing perspective

Most distributed training failures are expensive because they are discovered late. A user space preflight that validates scheduler topology, rank rendezvous, GPU affinity, and NCCL collectives provides a high value guardrail before production starts.

ai-infra-validator is a practical implementation of that guardrail. It is compact, transparent, and aligned with how real Slurm based AI clusters operate. For teams running multi node multi gpu training at scale, this kind of preflight should be a standard operational gate.

Distributing model weights to your AI cluster: a faster pre-flight on AKS and Slurm

pauledwards — Wed, 06 May 2026 18:30:00 GMT

Why this exists

You've provisioned a multi-node GPU cluster on Azure. NDR InfiniBand. The training script is ready. Before any GPU does any useful work, every node needs the same 400 GB model checkpoint on its local NVMe.

The naive approach — azcopy (or our Rust equivalent, azcp) on every node, in parallel — has three problems:

You pay Azure egress N times. Same bytes, same source account, N separate downloads. Microsoft's published scalability target for a standard storage account is 200 Gbps egress (region-dependent). It's a target, not a hard cap — our sharded download peaked at 236 Gb/s on a 16-node run — but if every node is hammering the same account independently you start seeing 503 ServerBusy well before the cluster fills up.
It's slower than your fabric should allow. Per-node Azure throughput tops out around 19-25 Gb/s on a default AKS pod (overlay network, single connection pool). Meanwhile the InfiniBand fabric between those same nodes is sitting idle at 400 Gb/s NDR.
Your job blocks on the slowest node. Distributed training won't start until rank 0 says "everyone has the data." With per-node downloads, that's whichever node had the unluckiest TCP retry.

You're paying for a 400 Gb/s fabric and using a 25 Gb/s pipe.

How azcp-cluster does it differently

The shape is obvious once you say it: download 1/N of the dataset per rank from Azure, then MPI_Ibcast the rest over the fabric. A sharded download across N nodes — each byte leaves Azure exactly once, instead of once per node — then fan out at fabric speed.

azcp-cluster is a small MPI binary that does exactly that:

Sharding uses deterministic LPT bin-packing — files sorted by size descending, greedily assigned to the least-loaded rank — so rank load spread is typically <2% even on lopsided file-size distributions (measured 1.3% on the DeepSeek-R1 checkpoint). Bcast is pipelined (non-blocking MPI_Ibcast with multiple chunks in flight per file, sized via --bcast-chunk / --bcast-pipeline) and writes are asynchronous (a dedicated writer thread per rank, decoupled from the MPI receive loop) so NVMe write speed and fabric receive overlap.

Measured on a 16-node Azure GB300 cluster, NDR 400 Gb/s, 4× HCA per node, 413 GiB / 524-file checkpoint to per-node NVMe (single example run; reproducer script tests/cluster_bench.sh):

Stage	Throughput	Wall-clock
[download] (Azure → ranks, aggregate)	236 Gb/s	14 s
[bcast] (per receiver, RDMA/UCX)	99.2 Gb/s	33 s
[total] end-to-end	—	48 s

The two stages run sequentially (bcast starts after download completes); the extra ~1 s in [total] is [list] + [diff] + [filelist] overhead. Per-node azcp for comparison: ~3-5 minutes wall-clock and 16× the egress bill (the whole 413 GiB transferred 16 times instead of once). It's not just the bill — it's also 16× the demand on the source storage account, which pushes you past the 200 Gbps target into 503 ServerBusy territory and the retry storm that comes with it. And that multiplier scales linearly with cluster size: at 64 nodes it's 64×, at 256 nodes it's 256×, while the broadcast version still pays Azure egress exactly once regardless of N.

Two things had to be true to get there: receivers parallelised the NVMe write path with multiple writer threads (default 2, flag --bcast-writers), and they opened destination files with O_DIRECT (default on, opt-out via --no-bcast-direct). With a single buffered writer the path collapses to ~28 Gb/s — page-cache contention, not the fabric, becomes the ceiling. Bypassing the page cache with O_DIRECT and parallelising across two writers per rank lifts that to 99 Gb/s; bumping to tmpfs (no disk in the path at all) shows the fabric ceiling is ~140 Gb/s on this hardware. Full sweep including a --verify end-to-end MD5 check is in docs/cluster-benchmarks.md.

After the download: the handoff

azcp-cluster finishes and every node has the full dataset on local NVMe. Now your training/inference job needs to start, on those same nodes, and read from those same disks. This is where the platform story diverges.

On Slurm, this is one sbatch with two srun steps. Done.

On AKS, you have to think about three things — storage backend, sequencing, and quota — and none of them have a single obvious answer yet. Both are below.

Practical: Slurm

Slurm makes this easy because the whole pipeline runs inside one sbatch allocation: nodes don't move between stages, per-node /mnt/nvme persists, and srun is a stage sequencer for free.

The full example is at examples/slurm/download-then-train.sbatch. The shape:

#SBATCH --nodes=4 #SBATCH --gres=gpu:8 #SBATCH --time=04:00:00 srun --ntasks-per-node=1 mkdir -p /mnt/nvme/dataset # Stage 1: download + bcast (azcp-cluster image) srun --mpi=pmix \ --container-image=/shared/images/azcp-cluster.sqsh \ --container-mounts=/dev/infiniband:/dev/infiniband,/mnt/nvme:/mnt/nvme \ azcp-cluster "$SOURCE_URL" /mnt/nvme/dataset \ --bcast-chunk 512M --bcast-pipeline 16 --compare size # Stage 2: training (different image, same allocation, same nodes) srun --container-image=/shared/images/training.sqsh \ --container-mounts=/mnt/nvme:/mnt/nvme \ torchrun --nnodes=$SLURM_NNODES --nproc-per-node=8 \ train.py --model-path /mnt/nvme/dataset

The snippet above is trimmed for readability. The full launcher (examples/slurm/download-then-train.sbatch) adds the UCX environment (UCX_TLS, UCX_NET_DEVICES, UCX_IB_GID_INDEX), --container-writable, --no-container-entrypoint, and the per-rank torchrun glue — drop those and you'll either fail to launch or silently fall back to TCP.

Two different containers, two different stages, one allocation. Slurm holds the node reservation across both srun calls, so the per-node NVMe state written by stage 1 is naturally available to stage 2 — no sentinel files, no priority classes, no operator coordination needed.

For sites without pyxis, swap --container-image= for apptainer exec; see examples/slurm/apptainer.sbatch.

Practical: AKS

Kubernetes has no Slurm-allocation analog out of the box — pods come and go, and the scheduler treats nodes as fungible resource buckets. So the question is: how do you keep the training job on the same nodes the download just populated?

The cleanest answer, by a wide margin, is to not separate them in the first place: run both phases inside one MPIJob, with one merged worker image that contains both azcp-cluster and your training stack. The launcher script becomes two mpirun calls, one after the other. The node allocation is held by the MPIJob across both phases — no cross-workload sequencing, no node-pinning glue, no extra controllers.

Adding azcp-cluster to your existing image

You almost certainly already have a curated training or inference image — NGC PyTorch, NGC TensorFlow, vLLM, your team's fine-tuning image, whatever. The merged image is just your image plus four COPY lines and one apt install:

FROM nvcr.io/nvidia/pytorch:24.10-py3 # or whatever you already use ARG AZCP_VERSION=v0.3.0 COPY --from=ghcr.io/edwardsp/azcp/azcp-cluster:${AZCP_VERSION} \ /opt/openmpi /opt/azcp/openmpi COPY --from=ghcr.io/edwardsp/azcp/azcp-cluster:${AZCP_VERSION} \ /opt/ucx /opt/azcp/ucx COPY --from=ghcr.io/edwardsp/azcp/azcp-cluster:${AZCP_VERSION} \ /usr/local/bin/azcp-cluster /usr/local/bin/ COPY --from=ghcr.io/edwardsp/azcp/azcp-cluster:${AZCP_VERSION} \ /usr/local/bin/azcp /usr/local/bin/ RUN apt-get update && apt-get install -y openssh-server && rm -rf /var/lib/apt/lists/*

Three notes on what's in those lines:

We copy azcp-cluster's bundled Open MPI 4.1.6 + UCX 1.15.0 under /opt/azcp/, not the default /opt/openmpi. NGC images already ship HPC-X at /opt/hpcx/; vLLM and other inference images sometimes bring their own MPI too. Putting ours at a separate path means we don't clash, and your training stack continues using whatever MPI it was built against.
The launcher script for stage 1 explicitly invokes /opt/azcp/openmpi/bin/mpirun, so phase 1 stays ABI-matched to the binary it was tested with.
mpi-operator's launcher SSHes into worker pods to bootstrap MPI; most training images ship openssh-client but not openssh-server, hence the one apt install. Skip it if your base image already has sshd.

That's the whole "merged image" cost. A couple of hundred MB of files copied (Open MPI + UCX + the two binaries), no Rust toolchain, no source rebuild, no rebase of your training image.

The two-phase launcher

The MPIJob launcher script runs both phases sequentially:

# Phase 1: download + broadcast mpirun -np "$N" -hostfile /etc/mpi/hostfile \ -mca pml ucx -x UCX_NET_DEVICES=mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1 \ azcp-cluster "$SOURCE_URL" /mnt/nvme/dataset \ --bcast-chunk 512M --bcast-pipeline 16 --compare size # Phase 2: training (mpirun launches torchrun on each node — NCCL, not # MPI, handles the actual training collectives) MASTER_ADDR=$(head -1 /etc/mpi/hostfile | awk '{print $1}') mpirun -np "$N" -hostfile /etc/mpi/hostfile \ -x MASTER_ADDR="$MASTER_ADDR" -x MASTER_PORT=29500 \ bash -c ' torchrun \ --nnodes='"$N"' --node_rank=$OMPI_COMM_WORLD_RANK \ --nproc_per_node=8 \ --master_addr=$MASTER_ADDR --master_port=$MASTER_PORT \ /opt/train.py --model-path /mnt/nvme/dataset '

The mpirun lines are trimmed to the load-bearing arguments. The real launcher in examples/aks/mpijob-download-then-train.yaml adds --allow-run-as-root, --prefix /opt/azcp/openmpi, -mca plm_rsh_agent ssh, -mca osc ucx, -mca routed direct, -mca coll_hcoll_enable 0, the full UCX environment (UCX_TLS=rc,sm,self, UCX_IB_GID_INDEX=3, …) and the worker pod's IPC_LOCK capability. Without those you either fail to launch or silently fall back to TCP — copy the full manifest, don't retype the snippet.

Phase 1 uses MPI for what MPI is good at — collective broadcast over RDMA. Phase 2 uses mpirun purely as a process launcher to start torchrun on every node with the right rank derived from $OMPI_COMM_WORLD_RANK; the actual training collectives go through NCCL, same as a plain torchrun Job. If your workload is genuinely MPI-native (Megatron-MPI, Horovod, DeepSpeed-MPI), drop the torchrun wrapper and mpirun your binary directly.

Authentication uses whatever azcp already supports (workload identity on AKS, IMDS on plain VMs, az-cli token, SAS, shared key); the MPIJob example wires up workload identity by setting AZURE_CLIENT_ID to a user-assigned managed identity that has been granted Storage Blob Data Reader on the source account. Adding the identity and the role assignment is a one-time setup step (az identity create + az role assignment create) and is the same work you'd do for any other AKS workload that talks to blob storage.

Dockerfile: examples/aks/Dockerfile.training. Storage is hostPath on the node's local NVMe — it persists naturally across the two phases (and across re-runs, where azcp-cluster --compare size skips already-present files for free).

When this isn't the right pattern

The merged image works for distributed training, fine-tuning, eval, batch inference, and anything else you'd normally launch with torchrun or mpirun. It does not fit two cases:

Long-running serving (vLLM, Ray, Deployment-shaped consumers). These load weights once and serve for hours or days; the per-node download cost is negligible amortized over the lifetime. Use a plain init container with azcp per pod and don't bother with broadcast.
The consumer must be a different CRD entirely, owned by another team, not changeable. Then you really do need two separate workloads, and you have a scheduling problem.

If you can't merge images

If the consumer genuinely can't share an image with azcp-cluster, the Kubernetes scheduler is suddenly responsible for two non-trivial things:

Same-nodes pinning. Kubernetes has no native "place this workload on the same nodes another workload ran on" primitive. You either dedicate a nodepool sized exactly to the job (so there's only one valid placement and the pool sits idle between runs) or you label nodes from the download pods, which needs cluster-level RBAC and per-run cleanup.

Gap-window starvation. When the download workload completes, its GPU and InfiniBand quota is released. On a busy multi-tenant cluster, another tenant's workload can race in and grab those nodes before your consumer is admitted — at which point the consumer queues indefinitely waiting for nodes it can never get back. Workload-level controllers like Kueue admit one workload at a time and don't reserve quota for "the next workload from the same submitter."

Reach for an initContainer first. Both problems vanish if you collapse the two workloads back into one Kubernetes object without merging images. Run a plain Indexed batch/v1.Job where each pod has an initContainer running azcp-cluster (broadcasting the dataset to per-node NVMe over RDMA) and a main container running the consumer against /mnt/nvme. Two different images, same pod, init-then-main ordering enforced structurally by Kubernetes. No node pinning needed (same pod ⇒ same node). No gap window (same pod ⇒ shared admission). No new CRDs. The cost is ~80 lines of bash to hand-roll the MPI bootstrap inside the initContainer (sshd, hostfile, mpirun) — see examples/aks/init-container-download-then-train.yaml for a validated reference. This is the second-best AKS option after the merged image, and it's the one to try before any of the heavier options below.

Heavier options if even that doesn't fit (multi-stage pipelines, long-running consumers, cross-team CRDs):

Argo Workflows can capture the download MPIJob's worker node names as an output parameter and inject them into the consumer's nodeAffinity. Doesn't fully close the gap window — pair with Kueue priorities to minimize it.
Volcano takes a different approach: a single Volcano Job CRD with internal task dependencies and gang scheduling, sequencing both stages in one admission unit. Cleanest model technically; not in the AKS managed-experience comfort zone.
JobSet + Kueue wraps both stages in one JobSet (single Kueue admission) with manual node-pinning via shared nodepool exhaustion. Requires installing JobSet (alpha CRD) on top of Kueue and hand-rolling an MPI bootstrap (sshd, hostfile, mpirun) inside an indexed Job, since JobSet wraps batch/v1 Jobs, not MPIJobs.

Each of these adds at least one extension to the cluster and substantially more YAML. For most distributed training and fine-tuning, the merged-image pattern is strictly simpler — the consumer image grows by four COPY lines and one apt install, and the entire scheduling problem disappears.

Try it

docker pull ghcr.io/edwardsp/azcp/azcp-cluster:v0.3.0

Multi-arch, linux/amd64 and linux/arm64. Source and full docs at github.com/edwardsp/azcp.

If you want to...	Start here
Run the standalone download on AKS (no downstream sequencing)	examples/aks/mpi-operator-job.yaml
Wire download + training together on AKS (merged image)	examples/aks/mpijob-download-then-train.yaml
Wire download + training together on AKS (separate images, no operator)	examples/aks/init-container-download-then-train.yaml
Run the standalone download on Slurm	examples/slurm/azcp-cluster.sbatch
Wire download + training together on Slurm	examples/slurm/download-then-train.sbatch
Reproduce the benchmark on your own cluster	tests/cluster_bench.sh + docs/cluster-benchmarks.md

What's next

A few things still on the list:

Multi-rail UCX bcast. The 99 Gb/s per-receiver number is the single-rail MPI_Ibcast ceiling; the GB300 nodes have 4 HCAs sitting there. Splitting the broadcast across rails (parallel Ibcasts on disjoint rail subsets, or a custom multi-tree implementation — not just a tuning flag) would push past the ~140 Gb/s tmpfs ceiling we see today.
Bcast tuning autodetect. Chunk/pipeline defaults are conservative; on NDR fabrics the right answer is --bcast-chunk 512M --bcast-pipeline 16. Probing at startup would be more friendly than asking users to read the tuning doc.
A first-class AKS sequencing story. Per the gap above.

Issues, PRs, and "hey we tried this and it broke on $cluster" reports all welcome at github.com/edwardsp/azcp.

Building resilient networks for AI supercomputers

jithinjose — Wed, 06 May 2026 16:25:26 GMT

By Valerie Cutts and Jithin Jose

Last fall we introduced Fairwater, the world’s most powerful AI datacenter. Delivering a system of this scale required rethinking how Azure designs supercomputers, especially the scale-out network. Today, we are sharing more about the networking innovations that have made Fairwater possible.

In this post, we share what’s unique about networking at extreme GPU scale and the system-level design choices required to enable large synchronous training jobs to run reliably, even during network failures. We are also publishing, in partnership with others, the open-source Multipath Reliable Connection (MRC) specification and software interfaces, and open-sourcing the associated libraries.

Fault-tolerant scale-out networking

At extreme scale, synchronous training amplifies the impact of routine network faults, turning packet drops, slow links, and partial failures into stalls, restarts, and wasted GPU time. As we describe in our MRC paper, the path forward is to treat failure as normal and design the network as an integrated system that:

Scales to 100K+ GPUs using a two level, multi-path topology to enable enough redundancy
Balances load evenly across the fabric to prevent congestion
Recovers predictably and gracefully during failures
Uses less power than three- or four-layer single-plane topologies

As outlined in the Multipath Reliable Connection Specification, Microsoft partnered with AMD, Broadcom, Intel, NVIDIA and OpenAI to jointly address this problem, focusing on changes to transport and network design needed to support training at extreme scale. Instead of relying on lossless fabrics and dynamic routing, we collectively designed, built, and deployed Multipath Reliable Connection (MRC), which draws upon lessons from the Ultra Ethernet Consortium (UEC), and paired it with a multiplane network topology, enabling reliable training jobs even when links, switches, or paths fail. The endpoint–driven transport created a simpler, more resilient network that delivers:

More resilient, predictable training at very large scale
Large training jobs continue making steady forward progress despite routine network faults, reducing stalls and restarts and improving time-to-train as cluster size increases.
Better utilization of expensive GPU infrastructure
By avoiding tail latency amplification and repeated recovery cycles, GPUs spend more time doing useful work instead of waiting on synchronization or replaying lost computation, improving overall efficiency and cost effectiveness.
Automatic adaptation at machine timescales
Failure detection, load balancing, and recovery happen fast enough to keep up with the rate and complexity of faults in 100K+ GPU systems, well beyond what manual intervention or control-plane convergence can achieve, allowing the system to remain stable as scale increases.

In the Fairwater supercomputer, enabling graceful degradation in the scale-out network improves training throughput versus traditional transports and architectures. In combination with a multi-plane topology design, MRC increases the time that installed NVIDIA GPUs perform useful computation.

A shift in philosophy: End-to-end control

The central design decision behind MRC is to shift responsibility for load balancing and failure handling from complex network switch control planes to the network endpoints with end-to-end controls. The network endpoint controls the path selection and can optimally use a set of paths based on feedback from the network.

MRC extends the RoCE Reliable Connection (RC) transport to support true multipath operation. Instead of binding a queue pair to a single path, MRC sprays packets across many paths simultaneously, making performance far less sensitive to any single slow or failed link. Several design elements are critical to enable end-to-end control.

Every packet carries enough information for the receiver to place it directly into memory, even if packets arrive out of order.
Selective acknowledgments enable rapid retransmission of only the packets that were lost.
Packet trimming signals network congestion swiftly without forcing full packet drops, enabling efficient congestion control.
MRC disables Priority Flow Control (PFC) entirely and runs Ethernet in best-effort mode, avoiding global pauses that can devastate tail latency or lead to fabric-wide deadlock behavior.
The system enables seamless self-recovery from network hardware failures.

The result is a transport protocol that expects loss, adapts quickly, and continues making progress even when parts of the fabric misbehave.

Rethinking topology: Multi‑plane design

Transport alone is not enough. To complement MRC, we implemented a two-tier, multiplane network topology in Fairwater using high-radix switches. Our network design splits each NIC into multiple lower speed ports (i.e. eight x 100 Gbps) and builds multiple parallel network planes. This multi-plane design enables a more compact topology, as opposed to a traditional three-tier Clos Network running at 800 Gbps/port. Our 2-layer multi-plane topology design offers several advantages:

Enables connecting 100K+ GPUs with just two tiers of network.
Lower latency since packets traverse fewer switches.
Reduced impact of network issues on overall job completion, while we see single switches connected to more servers, the individual impact is reduced by spreading the failure, decreasing the performance impact to the overall job.
Reduced hardware and power costs compared to designs with additional network layer, without compromising on GPU scale.
Most importantly, the network becomes more tolerant of partial failures, so jobs continue with slightly reduced bandwidth rather than failing outright.

Multiplane networks work efficiently only when traffic is evenly distributed across all planes and paths. This is where MRC’s packet spraying and path aware congestion response is essential.

Figure 1: Example of two-tier multi-plane topology using 512 x 100 GbE switches: 512 T0s x 256 NICs = 131,072 NICs

Static SRv6: Fewer moving parts, more predictability

In many data center networks, switches rely on Border Gateway Protocol (BGP) or other dynamic routing protocols. We removed dynamic routing from our design; instead, packets are source-routed using IPv6 Segment Routing (SRv6). Each packet encodes the end-to-end network path using compact microsegment IDs (uSIDs).

At first glance, static routing seems counterintuitive in failure-prone environments. At extreme scale, however, dynamic routing is more of a liability than an asset. Namely, if two or more switches try to reroute packets at the same time, network behavior becomes unpredictable and harder to diagnose. Interactions between adaptive routing and adaptive transport can be hard to resolve and harder to debug at larger scale.

MRC, on the other hand, handles path health and rapid failover at the transport layer. When static routing is used, it enables precise health feedback for each of the different network paths from that network endpoint. Because probe (test) packets follow the same paths as data packets, operators gain accurate, ground-truth insight into fabric health without depending on switch control planes, which are themselves a common source of failures.

Additionally, SRv6 routing allows network operators to utilize out-of-band monitoring frameworks to accurately identify link failures and device faults, which has been particularly valuable in managing large-scale AI clusters. Static SRv6 ensures paths are deterministic, making problems easier to reproduce, debug and, ultimately, more stable over time.

Failure as a normal operating condition

In production, failures are expected by design—link flaps, partial failures, and even switch reboots are routine at this scale. With MRC, many of these events no longer impact training workloads. Repair actions proceed in parallel, while MRC dynamically routes around failed paths. As repairs complete, MRC discovers and validates the restored paths before seamlessly reintegrating them—entirely transparent to the training application. In summary:

Systems degrade gracefully
Losing a NIC port reduces available bandwidth proportional to the lost port but does not crash jobs
Flapping T0–T1 links often go unnoticed by applications
Switches can be rebooted without coordinated drain or rerouting of the system

For massive scale training runs, this translates into higher effective uptime, fewer interrupted jobs, and more training throughput.

Figure 2: Bidirectional-bandwidth measurements with pt-pt RDMA Perftest while a T0 switch was taken down. Results indicate that the overall bandwidth dropped in proportion to the T0 switch bandwidth, but without failing the job

Figure 3: Bidirectional-bandwidth measurements with RDMA Perftest while T1 switch is failed and restored. Results indicate that no impact in performance as MRC was able to route around the bad switch

Measured results at scale

This is not just a thought exercise: Microsoft and OpenAI have both run extensive experiments and record-scale training jobs, showing only brief, bounded performance dips during significant network faults, followed by rapid recovery. Microbenchmarks demonstrate near line-rate bandwidth and predictable latency, even under injected loss. OpenAI describes their scale results in a recent blog post, consistent with what we observe. Taken together, multi-plane MRC with SRv6 delivers better load balancing with fewer queue pairs and substantially higher resilience to packet loss, enabling millions of networking links to connect hundreds of thousands of GPUs.

Figure 4: NCCL Send-Recv Benchmark Results with 42,020 GPUs each with 800 Gbps MRC NIC showing up to 92% of theoretical peak bandwidth for large message sizes

What this enables

Taken together, MRC, multiplane topologies, and static SRv6 form a coherent strategy for building AI supercomputer scale-out networks that keep large synchronous training jobs moving forward under real-world fault conditions. Instead of treating loss, link flaps and partial failures as events that trigger stalls or restarts, the system is designed to fail gracefully and reach 100K+ GPUs scale at high utilization. This design approach has been deployed in Fairwater and elsewhere to train state-of-the-art models, where the result is more predictable performance for large jobs with higher effective GPU utilization. The core takeaway is simple: by assuming failures will happen and, designing for them explicitly, events that would otherwise be catastrophic become minor, manageable perturbations.

Join us in advancing resilient AI infrastructure
To help the broader ecosystem adopt these capabilities, Microsoft is joining key partners in releasing the MRC specification to the Open Compute Project and open sourcing key components:

libMRC: MRC transport APIs
NCCL MRC plugin: enables NCCL to run over MRC transport
MRC shim library: enables compatible verbs applications to run over MRC with no code changes
MSCCL++ with MRC support: MSCCL++ library with MRC support
SONiC SRv6: enhance SRv6 with open NOS for high performance AI Ethernet

We encourage others to review these contributions to the public, share feedback, and ultimately adopt these capabilities within the broader ecosystem of AI networking products, infrastructures, and workloads.

Acknowledgements

Advancing AI at this scale requires collaboration across the industry. At Microsoft, we value our partnerships with AMD, Broadcom, Intel, NVIDIA, and OpenAI, and our shared commitment to continuing to evolve MRC alongside the broader community.

References:

Simplify troubleshooting at scale - Centralized Log Management for CycleCloud Workspace for Slurm

jesselopez — Tue, 31 Mar 2026 21:17:04 GMT

Training large AI models on hundreds or thousands of nodes introduces a critical operational challenge: when a distributed job fails, quickly identifying the root cause across scattered logs can become incredibly time-consuming. This manual process delays recovery and reduces cluster utilization. The ability to quickly parse centralized cluster logs from a single interface is critical to ensure job failure root cases are swiftly identified and mitigated to maintain high cluster utilization.

Solution Architecture

This is a turnkey, customizable log forwarding solution for CycleCloud Workspace for Slurm that centralizes all cluster logs into Azure Monitor Logs Analytic. The architecture uses Azure Monitor Agent (AMA) deployed on every VM and Virtual Machine Scale Set (VMSS) to stream logs defined by Data Collection Rules (DCR) to dedicated tables in a Log Analytics workspace where they can be queried from a single interface.

The turnkey solution captures three categories of logs essential for troubleshooting distributed workloads, but can be extended for any other logs:

Slurm logs including slurmctld, slurmd, etc., plus archived job artifacts (job submission scripts, environmental variables, stdout/stderr) collected via prolog/epilog scripts.

Infrastructure logs including those from CycleCloud including the CycleCloud Healthagent which automatically tests nodes for hardware health and draining nodes that fail tests.

Operation System logs from syslog and dmesg capturing kernel events, network state changes, and hardware issues.

Each log source flows through its own DCR into a dedicated table following a consistent schema. The solution automatically associates scheduler-specific DCRs with the Slurm scheduler node and compute-specific DCRs with compute nodes handling dynamic node scaling transparently.

The solution is purpose-built for CycleCloud Workspace for Slurm, but designed in a modular fashion to be easily extended for new data sources (i.e. new log formats) and processing (i.e. Data Collection Rules) to support log forwarding and analysis of other required logs.

Key Benefits

Time-series correlation: Azure Monitor's time-based indexing enables rapid identification of cascading failures. For example, trace a network carrier flap detected in syslog to corresponding slurmd communication errors to specific job failures all within seconds.

Centralized visibility: Query logs from thousands of nodes through a single interface instead of SSH-ing to individual machines. Correlate Slurm controller decisions with node-level errors and system events in one query.

Log persistence: Logs survive node deallocations and reimaging. Critical in cloud environments where compute nodes are ephemeral.

Powerful query language: KQL (Kusto Query Language) allows parsing raw logs into structured fields, filtering across multiple sources, and building operational dashboards. Example queries detect patterns like repeated job failures, network instability, or resource exhaustion.

Production-ready scalability: User-assigned managed identities automatically propagate to new VMSS instances, and DCR associations handle thousands of nodes without manual configuration.

Getting Started

The complete solution is available on GitHub (slurm-log-collection) with deployment scripts that:

Create all required Log Analytics tables

Deploys pre-configured DCRs for Slurm, CycleCloud, and OS logs

Automatically associate DCRs with scheduler and compute resources

After configuring environment variables and running the setup scripts, logs begin flowing to Azure Monitor and will populate within 15 minutes, but normal log ingestion latency is ~30s to 3 minutes. The repository includes sample KQL queries for common troubleshooting scenarios to accelerate time-to-resolution and to perform non-troubleshooting analysis of cluster usage.

Azure NCv6 Virtual Machines: Enhancements and GA Transition

Fernando_Aznar — Thu, 30 Apr 2026 18:00:24 GMT

NCv6 Virtual Machines are Azure's flexible, next generation platform enabling both leading-edge graphics and generative AI compute workloads. Featuring NVIDIA RTX PRO 6000 Blackwell Server Edition (BSE) GPUs, Intel Xeon™ 6 "Granite Rapids" 6900P series CPUs, and a suite of Microsoft Azure technologies, NCv6 VMs are available now in Preview.

Today, we are pleased to share a series of exciting updates coming soon to Azure NCv6 that will:

Enhance VM performance and capabilities
Provide more VM sizes for customers to "right size" their usage
Bring NCv6 to production readiness with a transition to General Availability, and
Expand accessibility across the global Azure cloud

New VM Sizes, Features, and Performance Enhancements

In the coming weeks, Azure will debut seven new NCv6-series VM sizes and two different sub-families for customers to choose from. The standout features introduced with the new VM sizes include:

🧩

Fractional GPU support, enabling graphics workload customers to deploy VMs with as little as 1/2 or 1/4 of a RTX PRO™ 6000. VMs with fractional GPU support also feature reduced vCPU, memory, SSD, and networking to help customers optimize costs and right size their VMs to their workloads.

⚡

Increased vCPU per VM size (e.g. 288 vCPU instead of 256) to provide more performance for high-end VDI workstations and better align with the Intel Xeon 6900P's triple compute tile architecture.

🛠️

General Purpose and Compute Optimized VM sizes. The former provides larger amounts of CPU memory for demanding generative AI inference and ISV CAD/CAE simulations, while the latter offers reduced memory to enable customers with less memory intensive workloads to cost optimize their deployments.

The new VM sizes will replace the existing three VM sizes offered in Preview, and be available as follows:

NCv6 - General Purpose VM sizes:

Size Name	vCPUs	Memory (GB)	Networking (Mb/s)	GPUs	GPU Mem (GB)	Temp Disk	NVMe Disk
Standard_NC36ds_xl_RTXPro6000_v6	36	132	22500	1/4	24	256	1600
Standard_NC72ds_xl_RTXPro6000_v6	72	264	45000	1/2	48	512	3200
Standard_NC144ds_xl_RTXPro6000_v6	144	516	90000	1	96	1024	6400
Standard_NC288ds_xl_RTXPro6000_v6	288	1032	180000	2	192	2048	12800
Standard_NC324ds_xl_RTXPro6000_v6	324	1284	180000	2	192	2048	12800

NCv6-Compute Optimized VM sizes:

Size Name	vCPUs	Memory (GB)	Networking (Mbps)	GPUs	GPU Mem (GB)	Temp Disk	NVMe Disk
Standard_NC24lds_xl_RTXPro6000_v6	24	72	22500	1/4	24	256	1600
Standard_NC36lds_xl_RTXPro6000_v6	36	72	22500	1/4	24	256	1600
Standard_NC72lds_xl_RTXPro6000_v6	72	132	45000	1/2	48	512	3200
Standard_NC144lds_xl_RTXPro6000_v6	144	264	90000	1	96	1024	6400
Standard_NC288lds_xl_RTXPro6000_v6	288	516	180000	2	192	2048	12800
Standard_NC324lds_xl_RTXPro6000_v6	324	648	180000	2	192	2048	12800

Note that, until the new VM sizes are available, Microsoft Learn resources will continue to reflect the currently offered VM sizes and technical specifications.

Transition to General Availability

In the coming weeks, Azure will transition NCv6-series from Preview to General Availability (GA) status. With this transition, NCv6 VMs will become covered by the Azure Service Level Agreement (SLA) and thus ready to support production-grade deployments by customers, partners, and service providers.

When the transition to NCv6 VMs occurs, they will be available in the Azure West US2 and Southeast Asia regions. Information on availability timing of additional regions is provided below.

Regional Expansion Across the Azure Cloud

At the beginning of Preview, NCv6 VMs debuted in the West US2 region. Since then, we have also added NCv6 VMs to the Southeast Asia region. Both regions will be part of the transition to GA status.

We are pleased to share that in the proceeding months covering Q3 of 2026, NCv6 VMs will also become available in the following Azure regions:

• East US

• West Europe

• East US 2

• North Europe

• South Central US

• Germany West Central

• West US

• Korea Central

Ready to build for the future with Azure NCv6?

NCv6 Virtual Machines are available now in Preview. Start your production-grade AI journey today and explore the next frontier of Azure AI infrastructure.

Join the Preview

AI Inferencing in Air-Gapped Environments

damocelj — Mon, 09 Mar 2026 09:18:31 GMT

If you had to point out the top trends of IT these days, two strong candidates would be Generative AI and Cybersecurity. Especially around the latter, sophistication, reach and volume of cyberattacks have seen significant increases in the last years, with added ingredients such as advanced persistent threats, state actors or “crime-as-a-service” providers.

Interestingly enough, both trends go hand in hand: Artificial Intelligence extracts value from your data, and cyber criminals are exactly after the same thing: your data. It is not surprising that organizations have taken steps to protect themselves against data theft or data exfiltration, as it is often described.

In this post we will explore how to deploy in a Kubernetes cluster a Hugging Face-hosted model and a NVIDIA NIM™ microservice, a prebuilt, optimized inference container for rapidly deploying the latest AI models, and at the same time protect your infrastructure against data theft. You can find more information about NVIDIA NIM here: https://developer.nvidia.com/nim.

We will outline the process for deploying a Kubernetes cluster in Azure with the highest level of network security to prevent data exfiltration, and we will also demonstrate the deployment of container images and required model parameters for both options

Why Kubernetes clusters

Unless you have been living under a rock, you are probably aware that Kubernetes has taken the IT world by storm, and the AI ecosystem is not an exception. Kubernetes makes it extremely easy to package and deploy applications over any infrastructure and hence it has become one the most popular platforms to run AI workloads, especially AI inferencing.

Azure Kubernetes Service (AKS) is an Azure service that makes it easy to run Kubernetes clusters in Azure. Over time, AKS has introduced multiple deployment options to meet increasingly stringent requirements, particularly around security. One such option is the private cluster, where no public IP addresses are assigned to the Kubernetes control plane or nodes.

To understand this evolution, let’s have a look at what a “public” AKS cluster looks like:

Figure 1- public AKS API enabled cluster

As the previous figure shows, there are multiple traffic flows that go over the public Internet:

In the bootstrap phase, the nodes get images from Microsoft Container Registry, as well as potentially from other repositories such as Ubuntu.
The Kubernetes administrator operates the cluster accessing the Kubernetes API provided by Microsoft with a public IP address.
When pulling container images, node clusters can get them from publicly available repositories such as Docker hub or Azure Container Registry (if configured to be publicly accessible).
Lastly, administrators are allowed to expose applications that run in the cluster via public IP addresses, so that users will access them over the Internet too.

The first evolution of this concept towards a more restrictive environment was a commonly used pattern consisting of a combination of private clusters (https://learn.microsoft.com/azure/aks/private-clusters) and Azure Firewall to limit egress traffic (https://learn.microsoft.com/azure/aks/limit-egress-traffic) and prevent data exfiltration.

In this model, there are no longer any inbound connections to the cluster:

Figure 2- AKS private cluster

The AKS API control plane is fully integrated in the virtual network.
Azure Container Registry and other Azure services such as Azure Storage or Azure Key Vault are also integrated with the virtual network through the Private Link technology (https://aka.ms/privatelink).

However, there are still outbound flows from the cluster nodes to the Internet, for example during the cluster creation process or the deployment of images stored in public repositories, which need to be explicitly allowed by the egress firewall.

Air-gapped clusters

It can be argued that using private clusters only provides security up to the robustness of your firewall ruleset: it essentially acts as a fail-open mechanism. If there’s a misconfiguration in the firewall rules, you may unintentionally allow data exfiltration or theft.

To address this, AKS offers an even greater degree of isolation with network-isolated clusters (https://learn.microsoft.com/azure/aks/concepts-network-isolated), where all outbound connections are completely blocked without the need of a firewall:

Figure 3- AKS isolated cluster

In this mode, AKS nodes are configured in a way so that no outbound flows to the Internet can exist.

If you are curious about what you need to do to make sure of that in Azure, here is the list:

No public Azure load balancer attached to the AKS nodes.
No NAT gateway attached to the AKS node subnet.
No public IP address attached to the AKS nodes.
The AKS node subnet configured for no default outbound access (https://learn.microsoft.com/azure/virtual-network/ip-services/default-outbound-access).

An important consideration is understanding how AKS nodes receive updates or how images are retrieved from public repositories (e.g. docker.io or nvcr.io). This is achieved through an Azure Container Registry feature known as “artifact cache”: https://learn.microsoft.com/azure/container-registry/artifact-cache-overview.

However, a challenge arises when considering Large Language Models (LLMs): LLM container images sourced from Hugging Face or NVIDIA (or any other source) typically include the inference runtime (for example vLLM) but not the model weights. Instead, model artifacts are downloaded dynamically when the container starts.

Consequently, Azure Container Registry cannot cache these assets. The question then becomes: how can these model weights be made available within an air-gapped Kubernetes environment?

The Model Weights Challenge

While the model weight (re) load on container startup is a flexible approach in connected environments, it fails in air-gapped clusters where outbound network access is blocked.

To address this, we consider two viable strategies:

Constructing a container image that includes all required components and pushing it to the container registry accessible by the isolated cluster
Pre-downloading model artifacts to a private file share connected to the virtual network of the isolated cluster, and accessing these resources as needed.

Both methods will be demonstrated in detail, but before proceeding, however, we will further outline the example scenario. To provide context aligned with current priorities among our financial clients and organizations operating within regulated sectors, this demonstration focuses on the process of model deployment and the configuration of an isolated cluster for LLM inferencing.

For inferencing we use Llama-3.1-8B-Instruct-FP8 served by vLLM, a high-performance inference runtime designed specifically for large language models. In simple terms, vLLM is responsible for efficiently loading the model onto the GPU and handling incoming inference requests with very low latency and high throughput. vLLM is typically packaged as a container image, which can be sourced either from Hugging Face or from NVIDIA (in our examples), the latter being highly optimized for NVIDIA GPUs and CUDA®. As described earlier, these images usually contain the inference runtime and dependencies, but not the model weights themselves.

Instead, the model weights and other model-specific artifacts are downloaded dynamically when the container starts, allowing the same container image to be reused across different models and versions while keeping the image size small and deployment flexible. This approach is not suitable in isolated AKS clusters, where network traffic flowing outside of the deployed virtual network is not permitted.

From an architectural perspective, model serving is only one part of the overall inferencing platform, and the design of the underlying GPU infrastructure plays a critical role-especially in isolated AKS clusters. In such environments, challenges are not limited to downloading model weights at container startup; for example, setting up the GPU node pool is another important consideration.

Traditionally, enabling GPUs on AKS requires installing the NVIDIA device plugin for Kubernetes as well as the NVIDIA GPU drivers, most commonly by deploying the NVIDIA GPU Operator, which takes care of both. While the device plugin itself can be installed relatively easily via the artifact cache of an attached container registry, driver installation is more involved, especially in air-gapped or isolated environments: https://learn.microsoft.com/azure/aks/use-nvidia-gpu. NVIDIA also provides detailed guidance on how to deploy the GPU Operator in such scenarios in their documentation: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator-air-gapped.html.

While the required procedures are clearly outlined and well documented, configuring the NVIDIA AKS GPU node pool within an air-gapped, isolated cluster continues to be a complex and time-consuming process.

Microsoft has recently unveiled a preview feature that allows users to create fully managed NVIDIA GPU node pools on AKS. With this option, all necessary NVIDIA components including drivers, device plugins, and other supporting software are pre-installed and maintained by Microsoft throughout their lifecycle.

This functionality is supported on isolated AKS clusters operating with Kubernetes v1.34.0 or later. It substantially decreases operational complexity, streamlining the deployment and maintenance of GPU-based AI inferencing solutions accelerated by NVIDIA in restricted environments https://learn.microsoft.com/en-us/azure/aks/aks-managed-gpu-nodes?tabs=add-ubuntu-gpu-node-pool

For example, this Azure CLI command would deploy a managed GPU AKS nodepool to an existing AKS cluster:

1 az aks nodepool add \

2 --resource-group MyResourceGroup \

3 --cluster-name MyAKSCluster \

4 --name gpunp \

5 --node-count 1 \

6 --node-vm-size <GPU_SKU> \

7 --node-taints sku=gpu:NoSchedule \

8 --enable-cluster-autoscaler \

9 --min-count 1 \

10 --max-count 3 \

11 --tags EnableManagedGPUExperience=true

Keep in mind that if you want complete control over driver versions and related settings, you should follow the guidelines for deploying the NVIDIA GPU Operator in an air-gapped environment. With the managed option, Microsoft takes care of maintaining driver versions for you.

Container and model weight deployment

With the managed GPU node pool set up, we can begin implementing both inferencing scenarios.

Scenario 1: Baking Model Weights into the Container Image

Let’s start with the first one, where the model weights are downloaded from Hugging Face, baked into a container image and then pushed to the attached Container Registry which is reachable from the isolated AKS cluster:

Figure 4- Baking Model Weights into Container Image

The easiest way to achieve this, is to trigger the container build directly from the container registry, which will take the local Dockerfile, pull the image and data needed from Hugging Face, and deploy and tag the backed image to the container registry.

Make sure you have acquired a Hugging Face API Key. If you are using a gated model, access must be requested before building the image.

1 az acr build \

2 --registry <ACR_NAME> \

3 --image llama3-vllm-fat:8b-instruct \

4 --build-arg HF_TOKEN=$HF_TOKEN \

5 .

Note: When you use the “az acr build” command instead of running docker build yourself, it automatically tags your image and pushes it to the Azure Container Registry.

Once this container image is available in the container registry, we can create a simple pod and an internal load balancer to expose the service endpoint to the user. The detailed instructions and code are available here: https://github.com/mocelj/aks-air-gap-vllm-deployment.

You can test the deployment by querying the external IP of the deployed service and interacting with the endpoint in an OpenAPI-compatible way . Note that since this is an isolated cluster, you need to connect to the cluster’s network via VPN or run the curl command in a pod inside of the cluster, see aks-air-gap-vllm-deployment/aks_isolated.sh at main · mocelj/aks-air-gap-vllm-deployment for more details about how to set up a point-to-site VPN in Azure. Here you can see how to get the service IP address and query the completions API:

1 # Get the Service IP

2 svc_ip=$(kubectl get svc vllm-llama3-8b -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

3 curl -X POST "http://${svc_ip}:8000/v1/chat/completions" \

4 -H "accept: application/json" \

5 -H "Content-Type: application/json" \

6 -d '{

7 "messages": [

8 {"role": "system", "content": "You are a polite and respectful chatbot."},

9 {"role": "user", "content": "Where should I go for lunch close to the Microsoft office in Pratteln?"}

10 ],

11 "model": "meta/llama3-8b-instruct",

12 "max_tokens": 512,

13 "top_p": 1,

14 "n": 1,

15 "stream": false,

16 "frequency_penalty": 0.0

17 }'

Scenario 2: Using a Shared File System for Model Artifacts

In the second scenario, where we are downloading the model weights and other artifacts used by the NVIDIA NIM to a shared NFS drive, we must follow a slightly different strategy.

Figure 5- Pre-download model weights to a private file share

For simplicity, we have used a virtual machine capable of downloading artifacts from the Internet and reaching the internal container registry as well as the shared NFS volume deployed in a virtual network. In this example, we have created a simple NFS share using Azure Files (see the Azure CLI code here to create the share and the endpoint): https://github.com/mocelj/aks-air-gap-vllm-deployment/blob/main/aks_isolated.sh#L352). For large scale inferencing scenarios, you might want to consider other storage options to ensure reasonable startup times, given the weights can be of considerable size.

To facilitate model deployment on NVIDIA A100 GPUs, we have provisioned a jump box equipped with the same GPU type. If you start with a fresh virtual machine, you may need to install the appropriate GPU driver as well as NVIDIA’s container runtime (https://developer.nvidia.com/container-runtime). You can alternatively deploy a Linux virtual machine using DSVM Linux images, where only the container runtime needs to be added to ensure readiness for operation.

To download the container image from nvcr.io we can leverage the caching rules in the container registry and pull the image via our connected container registry. Once the container image is pulled locally, we can download the model profile with the appropriate artifacts and copy everything to the shared folder, e.g. by using rsync.

The artifacts can be downloaded by using the Utilities for NVIDIA NIM for LLMs: https://docs.nvidia.com/nim/large-language-models/latest/utilities.html

1 docker run --rm \

2 --runtime=nvidia \

3 --gpus all \

4 -v $LOCAL_NIM_CACHE:/opt/nim/.cache \

5 -u $(id -u) \

6 -e NGC_API_KEY \

7 $TARGET_IMAGE \

8 download-to-cache

Important: The NVIDIA API key must not be included in the AKS deployment manifests, as otherwise it would trigger outbound network calls that will fail in air‑gapped environments. The key is only required on the jump box during the download of the model artefacts.

Since this shared folder is reachable from within the Jump box network and the isolated, air-gapped AKS cluster, the only thing we must do is pointing the NVIDIA NIM container to use the model weights found in the shared folder, and not to download it once the pod starts. It is important to note that the NVIDIA API key should not be part of the deployment script, since otherwise it will trigger an outbound connection to pull an image from the nvcr.io registry, which will fail in an air gapped environment.

The service can be tested in a similar way as before. First, we need to find out the external service IP, which we will get via “kubectl get svc vllm-nim-llama3-service -o wide” and then interact with the service in the same way as before, adjusting to the new service IP address.

A more detailed description of the implementation steps can be found in the attached repository: https://github.com/mocelj/aks-air-gap-vllm-deployment.

Summary

This document presents a practical guide for deploying LLM inferencing solutions in isolated Azure Kubernetes Service (AKS) clusters.

It outlines two deployment approaches: one where model weights and artifacts are pre-downloaded and stored in a shared folder accessible by both the jump box and cluster, and another using a shared NFS drive for storing downloaded resources.

Both strategies enable secure, air-gapped deployments without relying on outbound internet access. For step-by-step instructions and further technical details, consult the referenced GitHub repository https://github.com/mocelj/aks-air-gap-vllm-deployment.

For large‑scale or production deployments, more performant storage options—such as local NVMe or other high‑throughput solutions—can be explored; the services used in this guide are intentionally chosen to maximize clarity and reproducibility.

Microsoft at NVIDIA GTC 2026

Fernando_Aznar — Fri, 06 Mar 2026 23:44:35 GMT

Microsoft returns to NVIDIA GTC 2026 in San Jose with a strong presence across conference sessions, in‑booth theater talks, live demos, and executive‑level ancillary events. Together with NVIDIA and our partner ecosystem, Microsoft is showcasing how Azure AI infrastructure enables AI training, inference, and production at global scale. Visit us at Booth #521 to see the latest innovations in action and connect with Azure and NVIDIA experts.

Exclusive GTC Experiences

LEGO® Datacenter Model

Explore Azure AI infrastructure at the Park Container.

Candy Lounge

Visit the high-traffic candy wall for co-branded treats all day long.

Networking Lounge

Relax and recharge with comfy seating and vital charging options.

Outdoor Juice Truck

Free, refreshing beverages served during outdoor park hours.

Live from GTC: AI Podcast

Dayan Rodriguez

Corporate Vice President
Global Manufacturing
and Mobility

Alistair Spiers

General Manager
Azure Infrastructure

Live Special Feature

A conversation with Microsoft Azure

Listen & Subscribe:

aka.ms/GTC2026Podcast

Scan to Listen

Earned Conference Sessions

Don't miss these high-impact sessions where Microsoft and NVIDIA leaders discuss the future of AI factories and infrastructure.

Mon · Mar 16

5:00 PM

Drive Optimal Tokens per Watt on AI Infrastructure Using Benchmarking Recipes

Speakers: Paul Edwards, Emily Potyraj

Microsoft, NVIDIA

Tue · Mar 17

9:00 AM

Autonomous AI Factories: Technical Preview of Agent-Native Production

Speakers: JP Vasseur, César Martinez Spessot

NVIDIA, Microsoft Research

Tue · Mar 17

4:00 PM

The Road to Intelligent Mobility: Vehicle GenAI

Speakers: Raj Paul, Thomas Evans, Bryan Goodman

Microsoft, NVIDIA, Bosch

Wed · Mar 18

9:00 AM

Supercharging AI with Multi-Gigawatt AI Factories

Speakers: Gilad Shainer, Peter Salanki, Evan Burness

NVIDIA, CoreWeave, Meta, Microsoft

Daily Booth Theater Schedule

Visit the Microsoft Theater for lightning talks from engineering leaders and partners.

Monday, March 16

2:00 PM

BTH208 · NVIDIA

Accelerate AI Innovation on Azure with NVIDIA Run:ai — Rob Magno

2:30 PM

BTH202 · General Robotics

Models to Machines: Deploying Agentic AI in Real-World Robotics — Dinesh Narayanan

3:00 PM

BTH200 · Fractal Analytics

From Generalist to Enterprise-Ready: Fractal Builds Domain AI — C. Chaudhuri

3:30 PM

BTH109 · Microsoft

Agentic cloud ops - Smarter Operations with Azure Copilot — Jyoti Sharma

4:00 PM

BTH103 · Microsoft

Build a Deep Research Agent for Enterprise Data — D. Casati, A. Slutsky, H. Alkemade

4:30 PM

BTH205 · NetApp

Azure NetApp Files: Powering Your Data for AI Capabilities — Andy Chan

5:00 PM

BTH207 · NVIDIA

The Agentic Commerce Stack: Open Models on Azure — Antonio Martinez

5:30 PM

BTH217 · OPAQUE

Confidential AI on Azure Unlocks Sovereign AI at Scale — Aaron Fulkerson

6:00 PM

BTH218 · Simplismart

Making BYOC work at scale with modular inference — Amritanshu Jain

6:30 PM

Expo Reception

Tuesday, March 17

1:30 PM

BTH100 · Microsoft

From Open Weights to Enterprise Scale: Open-Source Models — Sharmila Chockalingam

2:00 PM

BTH212 · Personal AI

Unlocking the power of memory in Teams with Personal AI — Sam Harkness

2:30 PM

BTH111 · Microsoft / NVIDIA

Scalable LLM Inference on AKS Using NVIDIA Dynamo — Mohamad Al jazaery, Anton Slutsky

3:00 PM

BTH204 · Mistral AI

Innovate with Mistral AI on Microsoft Foundry — Ian Mathew

3:30 PM

BTH104 · Microsoft

GPU-Accelerated CFD at Scale: Star-CCM+ on Azure — Jason Scheffelmaer

4:00 PM

BTH206 · NeuBird AI

Agentic AI for Incident Response on Microsoft Azure — Grant Griffiths

4:30 PM

BTH101 · GitHub

Agentic DevOps: Evolving software with GitHub Copilot — Glenn Wester

5:00 PM

BTH209 · Rescale

Real-World AI Physics: GM & NVIDIA on Rescale — Dinal Perera

5:30 PM

BTH107 · Microsoft

Intro to LoRA Fine-Tuning on Azure — Christin Pohl

6:30 PM

Raffle

Wednesday, March 18

1:00 PM

BTH219 · VAST Data

Scaling AI Infrastructure on Azure with VAST Data — Jason Vallery

1:30 PM

BTH110 · Microsoft

Physical AI and Robotics: The Next Frontier — F. Miller, C. Souche, D. Narayanan

2:00 PM

BTH105 · Microsoft

Sovereign AI options with Azure Local — Kim Lam

2:30 PM

BTH108 · Microsoft

Automating HPC Workflows with Copilot Agents — Param Shah

3:00 PM

BTH102 · Microsoft

Trustworthy Multi-Agent Workflows with Microsoft Foundry — Brian Benz

4:00 PM

BTH106 · Microsoft

Scaling Enterprise AI on ARO with NVIDIA H100 & H200 — Lachie Evenson

4:30 PM

BTH211 · WEKA

Hybrid AI Data Orchestration with WEKA NeuralMesh™ — Desiree Campbell

5:00 PM

BTH202 · Hammerspace

NVIDIA AI Enterprise Software with NIM — Mike Bloom

5:30 PM

BTH203 · Kinaxis

Reimagining Global Supply Planning with Azure — Dane Henshall

6:00 PM

BTH214 · AT&T

Connected AI on Azure for Manufacturing — Brad Pritchett

6:30 PM

Raffle

Thursday, March 19

11:00 AM

BTH210 · Wandelbots

Physical AI: Powering Software-Defined Automation in Robotics — Marwin Kunz, Martin George

11:30 AM

Raffle

Explore Our Demo Pods

Visit the Microsoft booth to see our technology in action with live demonstrations across four dedicated pod areas.

POD 1

Azure AI Infrastructure

End‑to‑end AI infrastructure for training and inference at scale, featuring the latest NVIDIA GPU integrations on Azure.

POD 2

Microsoft Foundry

Our comprehensive platform for building, deploying, and operating agentic AI systems with enterprise reliability.

POD 3

Building AI Together

Showcasing joint Microsoft and NVIDIA solutions across diverse industries, from manufacturing to retail.

POD 4

Startups Powering AI

Discover how innovative startups are running next‑generation AI workloads on the Azure platform.

Ancillary Events & Networking

Join Microsoft leadership and our partner ecosystem at these curated networking experiences. Click the location to view on Bing Maps.

Sun · Mar 15

6:00

Microsoft for Startups Executive Leadership Dinner

📍 Morton’s Steakhouse, San Jose

Exclusive gathering for startup leaders and Microsoft executives.

Mon · Mar 16

1:30

Microsoft × NVIDIA Open Meet

📍 Signia by Hilton · International Suite

Strategic alignment session for Microsoft and NVIDIA executives.

Mon · Mar 16

7:30

Microsoft + NVIDIA Executive Dinner

📍 Il Fornaio, San Jose

Executive dinner for key customers and leadership teams.

Tue · Mar 17

11:00 AM

to 1:00 PM

Microsoft AI Luncheon: Research, Robotics, & Real‑World AI

📍 Signia by Hilton · International Suite

Invite-only: A curated executive lunch exploring the journey from AI research to physical enterprise deployments in robotics and manufacturing.

Tue · Mar 17

7:30

Networking in AI & Tech

📍 San Pedro Square Market

Community networking mixer for Microsoft teams, partners, and customers.

Wed · Mar 18

10:00 AM

to 1:00 PM

AI Innovator’s Circle Brunch: Powering Intelligent Systems Across the Ecosystem

📍 Il Fornaio, San Jose

Hosted by Microsoft & NVIDIA at GTC. Join us for an exclusive brunch and discussion on the intelligent ecosystem.

Azure Recognized as an NVIDIA Cloud Exemplar, Setting the Bar for AI Performance in the Cloud

Fernando_Aznar — Wed, 18 Feb 2026 22:31:25 GMT

As AI models continue to scale in size and complexity, cloud infrastructure must deliver more than theoretical peak performance. What matters in practice is reliable, end-to-end, workload-level AI performance—where compute, networking, system software, and optimization work together to deliver predictable, repeatable results at scale. This directly translates to business value: efficient full-stack infrastructure accelerates time-to-market, maximizes ROI on GPU and cloud investments, and enables organizations to scale AI from proof-of-concept to revenue-generating products with predictable economics.

Today, Microsoft is proud to share an important milestone in partnership with NVIDIA: Azure has been validated as an NVIDIA Exemplar Cloud, becoming the first cloud provider recognized for Exemplar-class AI performance aligned with GB300-class (Blackwell generation) systems.
This recognition builds on Azure’s previously validated Exemplar status for H100 training workloads and reflects NVIDIA’s confidence in Azure’s ability to extend that rigor and performance discipline into the next generation of AI platforms.

What Is NVIDIA Exemplar Cloud?

The NVIDIA Exemplar Cloud initiative celebrates cloud platforms that demonstrate robust end-to-end AI workload performance using NVIDIA’s Performance Benchmarking suite.

Rather than relying on synthetic microbenchmarks, Performance Benchmarking evaluates real AI training workloads using:

Large-scale LLM training scenarios
Production-grade software stacks
Optimized system and network configurations
Workload-centric metrics such as throughput and time-to-train

Achieving Exemplar validation signals that a provider can consistently deliver world-class AI performance in the cloud, showcasing that end users are getting optimal performance value by default.

Proven Exemplar Validation on H100

Azure’s Exemplar Cloud journey began with publicly shared benchmarking results for H100-based training workloads, where Azure ND GPU clusters demonstrated exemplar performance using NVIDIA Performance Benchmarking recipes.

Those results—published previously and validated through NVIDIA’s benchmarking framework—established a proven foundation of end-to-end AI performance for large-scale, production workloads running on Azure today.

Extending Exemplar-Class AI Performance to GB300-Class Platforms

Building on the rigor and learnings from H100 validation, Microsoft has now been recognized by NVIDIA as the first cloud provider to achieve Exemplar-class performance and readiness aligned with GB300-class systems.

This designation reflects NVIDIA’s assessment that the same principles applied to H100—including end-to-end system tuning, networking optimization, and software alignment—are being successfully carried forward into the Blackwell generation.

Rather than treating GB300 as a point solution, Azure approaches it as a continuation of a proven performance model: delivering consistent world-class AI performance in the cloud while preserving the flexibility, elasticity, and global scale customers expect.

What Enables Exemplar-Class AI Performance on Azure

Delivering Exemplar-class AI performance requires optimization across the full AI stack:

Infrastructure and Networking

High-performance Azure ND GPU clusters with NVIDIA InfiniBand
NUMA-aware CPU, GPU, and NIC alignment to minimize latency
Tuned NCCL communication paths for efficient multi-GPU scaling

Software and System Optimization

Tight integration with NVIDIA software, including Performance Benchmarking recipes and NVIDIA AI Enterprise
Parallelism strategies aligned with large-scale LLM training
Continuous tuning as models, workloads, and system architectures evolve

End-to-End Workload Focus

Measuring real training performance, not isolated component metrics
Driving repeatable improvements in application-level throughput and efficiency
Closing the performance gap between cloud and on-premises systems—without sacrificing manageability

Together, these capabilities enabled Azure to deliver consistent Exemplar-class AI performance across generations of NVIDIA platforms.

What This Means for Customers

For customers training and deploying advanced AI models, this milestone delivers clear benefits:

World-class AI performance in a fully managed cloud environment
Predictable scaling from small clusters to thousands of GPUs
Faster time to train and improved performance per dollar
Confidence that Azure is ready for Blackwell-class and GB300-class AI workloads

As AI workloads become more complex and reasoning-heavy, infrastructure performance increasingly determines outcomes. Azure’s NVIDIA Cloud Exemplar recognition provides a clear signal: customers can build and scale next-generation AI systems on Azure without compromising on performance.

Learn More

DGX Cloud Benchmarking on Azure
DGX Cloud Benchmarking on Azure | Microsoft Community Hub

Centralized cluster performance metrics with ReFrame HPC and Azure Log Analytics

jimpaine — Fri, 06 Feb 2026 09:37:24 GMT

Imagine having several clusters across different environments (dev, test and prod) or planning a migration between PBS and Slurm or porting codes to a different system. They can all seem like daunting tasks.

This is where the combination of ReFrame HPC, a powerful and feature rich testing framework, and Azure Log Analytics can help improve confidence and assurance in the performance and accuracy of a system.

Here we will look at how to configure ReFrame HPC specifically for Azure: Deploying the required Azure resources, running a test and capturing the results in Log Analytics for analysis.

Deploying the required Azure Resources

Firstly, deploy the required resources in Azure by using this bicep from GitHub. The deployment includes the creation and configuration of everything required for ReFrame HPC. These resources include a data collection endpoint, a data collection rule and a log analytics workspace.

Azure icons for a Data Collection Endpoint, Data Collection Rule with an arrow pointing from them to the icon for Log Analytics Workspace.

The structure of the endpoint that is needed later is complex, but the bicep generates it and outputs it at the end so make sure to caputure it now.

Running ior via ReFrame HPC

For the purpose of demonstrating a running test and capturing the results in Azure from start to finish, here is a simple ior test which will run both a read and a write operation against the shared storage.

import reframe as rfm import reframe.utility.sanity as sn @rfm.simple_test class SimplePerfTest(rfm.RunOnlyRegressionTest): valid_systems = ["*"] valid_prog_environs = ["+ior"] executable = 'ior' executable_opts = [ '-a POSIX -w -r -C -e -g -F -b 2M -t 2M -s 25600 -o /data/demo/test.bin -D 300' ] reference = { 'tst:hbv4': { 'write_bandwidth_mib': (500, -0.05, 0.1, 'MiB/s'), 'read_bandwidth_mib': (350, -0.05, 0.5, 'MiB/s'), } } @sanity_function def validate_run(self): return sn.assert_found(r'Summary of all tests:', self.stdout) @performance_function('MiB/s') def write_bandwidth_mib(self): return sn.extractsingle(r'^write\s+([0-9]+\.?[0-9]*)', self.stdout, 1, float) @performance_function('MiB/s') def read_bandwidth_mib(self): return sn.extractsingle(r'^read\s+([0-9]+\.?[0-9]*)', self.stdout, 1, float)

Test explanation

Set the binary to be executed to ior, along with its arguments.

executable = 'ior' executable_opts = [ '-a POSIX -w -r -C -e -g -F -b 2M -t 2M -s 25600 -o /data/demo/test.bin -D 300' ]

Specify which systems the test should run on. In this case, any system/cluster which is known to have ior available will be selected. Look at the ReFrame HPC documentation to get a better understanding of the options available for use.

valid_systems = ["*"] valid_prog_environs = ["+ior"]

Verify the stdout of the job by searching for a specific value to assert that it ran successfully.

@sanity_function def validate_run(self): return sn.assert_found(r'Summary of all tests:', self.stdout)

If the sanity function passed it will then extract the performance metrics from the stdout of the job. The naming of the methods is important, as they will be stored in the results later.

@performance_function('MiB/s') def write_bandwidth_mib(self): return sn.extractsingle(r'^write\s+([0-9]+\.?[0-9]*)', self.stdout, 1, float) @performance_function('MiB/s') def read_bandwidth_mib(self): return sn.extractsingle(r'^read\s+([0-9]+\.?[0-9]*)', self.stdout, 1, float)

Performance references are used to determine if the current cluster has met the requirement or not. It also allows margins to be specified in either direction.

reference = { 'tst:hbv4': { 'write_bandwidth_mib': (500, -0.05, 0.1, 'MiB/s'), 'read_bandwidth_mib': (350, -0.05, 0.5, 'MiB/s'), } }

ReFrame HPC Configuration

The ReFrame HPC configuration is key to determine how and where the test will run. It is also where the logic allowing Reframe HPC to use Azure for centralized logging will be defined. The full configuration file is vast and is covered in detail within the ReFrame HPC documentation. For the purpose of this test an example can be found on GitHub. Below is a breakdown of the key parts that allow Reframe HPC to push its results into Azure Log Analytics.

Logging Handler

The most important part of this configuration is the logging section, without it ReFrame HPC will not attempt to log the results. A handler_perflog of type httpjson is added to enable the logs to be sent to a HTTP endpoint with specific values which our covered below.

'logging': [ { 'perflog_multiline': True, 'handlers_perflog': [ { 'type': 'httpjson', 'url': 'REDACTED', 'level': 'info', 'debug': False, 'extra_headers': {'Authorization': f'Bearer {_get_token()}'}, 'extras': { 'TimeGenerated': f'{datetime.now(timezone.utc).isoformat()}', 'facility': 'reframe', 'reframe_azure_data_version': '1.0', }, 'ignore_keys': ['check_perfvalues'], 'json_formatter': _format_record } ] }

Multiline Perflog

To ensure this works with Azure, enable perflog_multiline. This will ensure a single record per metric is sent to Log Analytics. This is the cleanest way to output the results. Having this set to False will move the metric names into column names, which means that the schema will be different for each test and will become hard to maintain.

Extra Headers

A bearer token is required to authenticate the request. ReFrame HPC allows the adding of headers via the extra_headers property and a simple Python function, which obtains a scoped token that can be appended to the additional header.

def _get_token(scope='https://monitor.azure.com/.default') -> str: credential = DefaultAzureCredential() token = credential.get_token(scope) return token.token

Url Structure

The url can be found in the output of the bicep which was run previously. It can also be obtained via the portal. Here is the structure of the url for reference.

'${dce.properties.logsIngestion.endpoint}/dataCollectionRules/${dcr.properties.immutableId}/streams/Custom-${table.name}?api-version=2023-01-01'

json Formatter

A small work around is needed as the Data Collection Rule expects an array of items and ReFrame HPC outputs a single record. To resolve this another Python function can be used which simply wraps the record up in an array. In this example it also tidys up and removes some items that are not required and would cause issues with the json serialization.

def _format_record(record, extras, ignore_keys): data = {} for attr, val in record.__dict__.items(): if attr in ignore_keys or attr.startswith('_'): continue data[attr] = val data.update(extras) return json.dumps([data])

Running the Test

Now that the infrastructure has been deployed, the test has been defined and is correctly configured, we can run the test.

Start by logging in. Here I am using the managed identity of the node, but User auth and User Assigned Managed Identities are also supported.

$ az login --identity

ReFrame HPC can be installed via Spack or Python and, while I am using Spack for packages on the cluster, I find the simplest approach is to activate a Python environment and install ReFrame HPC along with test specfic Python dependencies.

$ python3 -m venv .venv $ . .venv/bin/activate $ python -m pip install -U pip $ pip install -r requirements.txt

Now using the ReFrame HPC cli, the test can be run using the configuration file and the test file.

$ reframe -C config.py -c simple_perf.py --performance-report -r

ReFrame HPC will now run the test against the system/cluster defined in the configuration. For this example it is a Slurm cluster on a partition of HBv4 nodes and running squeue clarifys that.

$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 955 hbv4 rfm_Simp jim.pain R 0:28 1 tst4-hbv4-97

Results

And there we have it, results are now appearing in Azure! From here we can use kql to query and filter the results. This is just a subset of the values available but the dataset is vast and includes a huge range of values that are extremely helpful.

Summary

By standardizing on the combination of ReFrame HPC and Azure Log Analytics for testing and reporting of performance data across our clusters, whether Slurm based, Azure CycleCloud or existing on-prem clusters, you can gain unprecendented visibility and confidence in the systems you manage and the codes you deploy that were previously hard to obtain. Enabling the potential for:

🔎Fast cross-cluster comparisions
📈Trend analysis over long running periods
📊Standardized metrics regardless of scheduler or system
☁️Unified monitoring and reporting across clusters

ReFrame HPC is suitable for a wide range of testing, so if testing is something you have been looking to implement, take a look at ReFrame HPC

Scaling physics-based digital twins: Neural Concept on Azure delivers a New Record in Industrial AI

lmiroslaw — Mon, 12 Jan 2026 12:10:23 GMT

Automotive Design and the DrivAerNet++ Benchmark

In automotive design, external aerodynamics have a direct impact on performance, energy efficiency, and development cost. Even small reductions in drag can translate into significant fuel savings or extended EV range. As development timelines accelerate, engineering teams increasingly rely on data-driven methods to augment or replace traditional CFD workflows.

MIT’s DrivAerNet++ dataset is the largest open multimodal dataset for automotive aerodynamics, offering a large-scale benchmark for evaluating learning-based approaches that capture the physical signals required by engineers. It includes 8,000 vehicle geometries across 3 variants (fastback, notchback and estate-back) and aggregates 39 TB of high-fidelity CFD outputs such as surface pressure, wall shear stress, volumetric flow fields, and drag coefficients.

Benchmark Highlights

Neural Concept trained its geometry-native Geometric Regressor, designed to handle any type of engineering data. The benchmark was executed on Azure HPC infrastructure to evaluate the capabilities of the geometry-native platform under transparent, scalable, and fully reproducible conditions.

Surface pressure: Lowest prediction error recorded on the benchmark, revealing where high- and low-pressure zones form.
Wall shear stress: Outperforming all competing methods to detect flow attachment and separation for drag and stability control.
Volumetric velocity field: More than 50% lower error than previous best, capturing full flow structure for wake stability analysis.
Drag coefficient Cd: R² of 0.978 on the test set, accurate enough for early design screening without full CFD runs.
Dataset Scale and Ingestion: 39 TB of data was ingested into Neural Concept’s platform through a parallel conversion task with 128 workers and 5 GB RAM each that finished in about 1 hour and produced a compact 3 TB dataset in the platform’s native format.
Data Pre Processing: Pre-processing the dataset required both large-scale parallelization and the application of our domain-specific best practices for handling external aerodynamics workflows.
Model Training and Deployment: Training completed in 24 hours on 4 A100 GPUs, with the best model obtained after 16 hours. The final model is compact and real-time predictions can be served on a single 16 GB GPU for industrial use.

Neural Concept outperformed all other competing methods, achieving state-of-the-art performance prediction on all metrics and physical quantities within a week:

“Neural Concept’s breakthrough demonstrates the power of combining advanced AI with the scalability of Microsoft Azure,” said Jack Kabat, Partner, Azure HPC and AI Infrastructure Products, Microsoft. “By running training and deployment on Azure’s high-performance infrastructure — specifically the NC A100 Virtual Machine— Neural Concept was able to transform 39 terabytes of data into a production-ready workflow in just one week. This shows how Azure accelerates innovation and helps automotive manufacturers bring better products to market faster.”

For additional benchmark metrics and comparisons, please refer to the Detailed Quantitative Results section at the end of the article.

From State-Of-The-Art Benchmark Accuracy to Proven Industrial Impact

Model accuracy alone is necessary, but not sufficient for industrial impact. Transformative gains at scale and over time are only revealed once high-performing models are deployed into maintainable and repeatable workflows across organizations.

Customers using Neural Concept’s platform have achieved:

30% shorter design cycles
$20M in savings on a 100,000-unit vehicle program

These outcomes fundamentally result from a transformed, systematic approach to design, unlocking better and faster data-driven decisions. The Design Lab interface, described in the next section, is at the core of this transformation.

Within Neural Concept’s ecosystem, validated geometry and physics models can be deployed directly into the Design Lab - a collaborative environment where aerodynamicists and designers evaluate concepts in real time. AI copilots provide instant performance feedback, geometry-aware improvement suggestions, and live KPI updates, effectively reconnecting aerodynamic analysis with the pace of modern vehicle design.

CES 2026: See how OEMs are transforming product development with Engineering Intelligence

Neural Concept and Microsoft will showcase how AI-native aerodynamic workflows can reshape vehicle development — from real-time design exploration to enterprise-scale deployment. Visit the Microsoft booth to see DrivAerNet++ running on Azure HPC and meet the teams shaping the future of automotive engineering.

Visit Microsoft Booth to find out more

Neural Concept’s executive team will also be at CES to share flagship results achieved by leading OEMs and Tier-1 suppliers already using the platform in production. Learn more on: https://www.neuralconcept.com/ces-2026

Credits
Microsoft: Hugo Meiland (Principal Program Manager), Guy Bursell (Director Business Strategy, Manufacturing), Fernando Aznar Cornejo (Product Marketing Manager) and Dr. Lukasz Miroslaw (Sr. Industry Advisor)

Neural Concept: Theophile Allard (CTO), Benoit Guillard (Senior ML Research Scientist), Alexander Gorgin (Product Marketing Engineer), Konstantinos Samaras-Tsakiris (Software Engineer)

Detailed Quantitative Results

In the sections that follow, we share the results obtained by applying Neural Concept’s aerodynamics predictive model training template to Drivaernet++.

We evaluated our model’s prediction errors using the official train/test split and the standard evaluation strategy. For comparison, metrics from other methods were taken from the public leaderboard. We reported both Mean Squared Error (MSE) and Mean Absolute Error (MAE) to quantify prediction accuracy. Lower values for either metric indicate closer agreement with the ground truth simulations, meaning better predictions.

1. Surface Field Predictions: Pressure and Wall Shear Stress

We began by evaluating predictions for the two physical quantities defined on the vehicle surface.

Surface Pressure

The Geometric Regressor achieved substantially better performance than all existing methods in predicting surface pressure distribution.

Rank	Deep Learning Model	*MSE (10-2, lower = better)**	*MAE (10-1, lower = better)**
#1	Neural Concept	3.98	1.08
#2	GAOT (May 2025)	4.94	1.10
#3	FIGConvNet (February 2025)	4.99	1.22
#4	TripNet (March 2025)	5.14	1.25
#5	RegDGCNN (June 2024)	8.29	1.61

Table 1: Neural Concept’s Geometric Regressor predicts surface pressure more accurately than previously published state-of-the-art methods. The dates indicate when the competing model architectures were published.

Figure 1: Side-by-side comparison of the ground truth pressure field (left), Neural Concept model’s prediction (middle), and the corresponding error for a representative test sample (right).

Wall Shear Stress

Similarly, the model delivered top-tier results, outperforming all competing methods.

Rank	Deep Learning Model	*MSE (10^-2, lower = better)**	*MAE (10^-1, lower = better)**
#1	Neural Concept	7.80	1.44
#2	GAOT (May 2025)	8.74	1.57
#3	TripNet (March 2025)	9.52	2.15
#4	FIGConvNet (Feb. 2025)	9.86	2.22
#5	RegDGCNN (June 2024)	13.82	3.64

Table 2: Neural Concept’s Geometric Regressor predicts wall shear stress more accurately than previously published state-of-the-art methods.

Figure 2: Side-by-side comparison of the ground truth magnitude of the wall shear stress, Neural Concept model’s prediction, and the corresponding error for a representative test sample.

Across both surface fields (pressure and wall shear stress), the Geometric Regressor achieved the lowest MSE and MAE by a clear margin. The baseline methods represent several high-quality and recent academic work (the earliest being from June 2024), yet our architecture established a new state-of-the-art in predictive performance.

2. Volumetric Predictions: Velocity

Beyond surface quantities, DrivAerNet++ provides 3D velocity fields in the flow volume surrounding the vehicle, which we also predicted using the Geometric Regressor.

Rank	Deep Learning Model	MSE (lower = better)	*MAE (10^-1, lower = better)**
#1	Neural Concept	3.11	9.22
#2	TripNet (March 2025)	6.71	15.2

Table 3: Neural Concept’s Geometric Regressor predicts velocity more accurately than the previously published state-of-the-art method.

The illustration below shows the velocity magnitude for two test samples. Note that only a single 2D slice of the 3D volumetric domain is shown here, focusing on the wake region behind the car. In practice, the network predicts velocity at any location within the full 3D domain, not just on this slice.

Figure 3: Velocity magnitude for two test samples, arranged in two columns (left and right). For each sample, the top row displays the simulated velocity field, the middle row shows the prediction from the network, and the bottom row presents the error between the two.

3. Scalar Predictions: Drag Coefficient

The drag coefficient (Cd) is the most critical parameter in automotive aerodynamics, as reducing it directly translates to lower fuel consumption in combustion vehicles and increased range in electric vehicles. Using the same underlying architecture, our model achieved state-of-the-art performance in Cd prediction.

In addition to MSE and MAE, we reported the Maximum Absolute Error (Max AE) to reflect worst-case accuracy. We also included the Coefficient of Determination (R² score), which measures the proportion of variance explained by the model. An R² value of 1 indicates a perfect fit to the target data.

Rank	Deep Learning Model	*MSE (1e-5)**	*MAE (1e-3)**	*Max AE (1e-2)**	R²
#1	Neural Concept	0.8	2.22	1.13	0.978
#2	TripNet	9.1	7.19	7.70	0.957
#3	PointNet	14.9	9.60	12.45	0.643
#4	RegDGCNN	14.2	9.31	12.79	0.641
#5	GCNN	17.1	10.43	15.03	0.596

On the official split, the model shows tight agreement with CFD (R² of 0.978) across the test set, which is sufficient for early design screening where engineers need to rank variants confidently and spot meaningful gains without running full simulations for every change.

4. Compute Efficiency and Azure HPC&AI Collaboration

Executing the full DrivAerNet++ benchmark at industrial scale required Neural Concept’s full software and infrastructure stack combined with seamless cloud integration on Microsoft Azure to dynamically scale computing resources on demand. The entire pipeline runs natively on Microsoft Azure and can scale within minutes, allowing us to process new industrial datasets that contain thousands of geometries without complex capacity planning.

Dataset Scale and Ingestion

DrivAerNet++ dataset contains 8000 car designs along with their corresponding CFD simulations. The raw dataset occupies approximately 39TB of storage. Generating the simulations required a total of about 3 million CPU hours by MIT’s DeCoDE Lab.

Ingestion into Neural Concept’s platform is the first step of the pipeline.

To convert the raw data into the platform’s native format, we use a Conversion task that transforms raw files into the platform’s optimized native format.
This task was parallelized with 128 workers; each allocated 5 GB of RAM.

As a result, the entire conversion process was completed in approximately one hour only. After converting the relevant data (car geometry, wall shear stress, pressure, and velocity), the full dataset occupies approximately 3 TB in Neural Concept’s native format.

Data Pre-Processing

Pre-processing the dataset required both large-scale parallelization and the application of our domain-specific best practices. During this phase, workloads were distributed across multiple compute nodes with peak memory usage reaching approximately 1.5 TB of RAM.

The pre-processing pipeline consists of two main stages. In the first stage, we repaired the car meshes and pre-computed geometric features needed for training. The second stage involved filtering the volumetric domain and re-sampling points to follow a spatial distribution that is more efficient for training our deep learning model.

We scaled the compute resources so that each of the two stages in the pipeline completes in 1 to 3 hours when processing the full dataset. The first stage is the most computationally intensive. To handle it efficiently, we parallelized the task across 256 independent workers, each allocated 6 GB of RAM.

Model Training and Deployment

While we use state-of-the-art hardware for training, our performance gains come primarily from model design. Once trained, the model remains lightweight and cost-effective to run.

Training was performed on Azure Standard_NC96ads_A100_v4 node, which provided access to four A100 GPUs, each with 80 GB of memory.
The model was trained for approximately 24 hours.

Neural Concept’s Geometric Regressor achieved the best reported performance on the official benchmark for surface pressure, wall shear stress, volumetric velocity and drag prediction.

mpi-stage: High-Performance File Distribution for HPC Clusters

pauledwards — Fri, 09 Jan 2026 10:24:34 GMT

When running containerized workloads on HPC clusters, one of the first problems you hit is getting container images onto the nodes quickly and repeatably. A .sqsh is a Squashfs image (commonly used by container runtimes on HPC). In some environments you can run a Squashfs image directly from shared storage, but at scale that often turns the shared filesystem into a hot spot.

Copying the image to local NVMe keeps startup time predictable and avoids hundreds of nodes hammering the same source during job launch.

In this post, I'll introduce mpi-stage, a lightweight tool that uses MPI broadcasts to distribute large files across cluster nodes at speeds that can saturate the backend network.

The Problem: Staging Files at Scale

On an Azure CycleCloud Workspace for Slurm cluster with GB300 GPU nodes, I needed to stage a large Squashfs container image from shared storage onto each node's local NVMe storage before launching training jobs.

At small scale you can often get away with ad-hoc copies, but once hundreds of nodes are all trying to read the same source file, the shared source filesystem quickly becomes the bottleneck.

I tried several approaches:

Attempt 1: Slurm's sbcast

Slurm's built-in sbcast seemed like the natural choice. In my quick testing it was slower than I wanted, and the overwrite/skip-existing behavior didn't match the "fast no-op if already present" workflow I was after. I didn't spend much time exploring all the configuration options before moving on.

Attempt 2: Shell Script Fan-Out

I wrote a shell script using a tree-based fan-out approach: copy to N nodes, then each of those copies to N more, and so on. This worked and scaled reasonably, but had some drawbacks:

Multiple stages: The script required orchestrating multiple rounds of copy commands, adding complexity
Source filesystem stress: Even with fan-out, the initial copies still hit the source filesystem simultaneously — a fan-out of 4 meant 4 nodes competing for source bandwidth
Frontend network: Copies went over the Ethernet network by default — I could have configured IPoIB, but that added more setup

The Solution: MPI Broadcasts

The key insight was that MPI's broadcast primitive (MPI_Bcast) is specifically optimized for one-to-many data distribution. Modern MPI implementations like HPC-X use tree-based algorithms that efficiently utilize the high-bandwidth, low-latency InfiniBand network.

With mpi-stage:

Single source read: Only one node reads from the source filesystem
Backend network utilization: Data flows over InfiniBand using optimized MPI collectives
Intelligent skipping: Nodes that already have the file (verified by size or checksum) skip the copy entirely

Combined, this keeps the shared source (NFS, Lustre, blobfuse, etc.) from being hammered by many concurrent readers while still taking full advantage of the backend fabric.

How It Works

mpi-stage is designed around a simple workflow:

The source node reads the file in chunks and streams each chunk via MPI_Bcast. Destination nodes write each chunk to local storage immediately upon receipt. This streaming approach means the entire file never needs to fit in memory — only a small buffer is required.

Key Features

Pre-copy Validation

Before any data is transferred, each node checks if the destination file already exists and matches the source. You can choose between:

Size check (default): Fast comparison of file sizes—sufficient for most use cases
Checksum: Stronger validation, but requires reading the full file and is therefore slower

If all nodes already have the correct file, mpi-stage completes in milliseconds with no data transfer.

Double-Buffered Transfers

The implementation uses double-buffered, chunked transfers to overlap network communication with disk I/O. While one buffer is being broadcast, the next chunk is being read from the source.

Post-copy Validation

Optionally verify that all nodes received the file correctly after the copy completes.

Single-Writer Per Node

The tool enforces one MPI rank per node to prevent filesystem contention and ensure predictable performance.

Real-World Performance

In one run using 156 GPU nodes, distributing a container image achieved approximately 3 GB/s effective distribution rate (file_size/time), completing in just over 5 seconds:

[0] Copy required: yes [0] Starting copy phase (source writes: yes) [0] Copy complete, Bandwidth: 3007.14 MB/s [0] Post-validation complete [0] Timings (s): Topology check: 5.22463 Source metadata: 0.00803746 Pre-validation: 0.0046786 Copy phase: 5.21189 Post-validation: 2.2944e-05 Total time: 5.2563

Because every node writes the file to its own local NVMe, the cumulative write rate across the cluster is roughly this number times the node count: ~3 GB/s × 156 ≈ ~468 GB/s of total local writes.

Workflow: Container Image Distribution

The primary use case is distributing Squashfs images to local NVMe before launching containerized workloads. Run mpi-stage as a job step before your main application:

#!/bin/bash #SBATCH --job-name=my-training-job #SBATCH --ntasks-per-node=1 #SBATCH --exclusive # Stage the container image srun --mpi=pmix ./mpi_stage \ --source /shared/images/pytorch.sqsh \ --dest /nvme/images/pytorch.sqsh \ --pre-validate size \ --verbose # Run the actual job (from local NVMe - much faster!) srun --container-image=/nvme/images/pytorch.sqsh ...

mpi-stage will create the destination directory if it doesn't exist.

If your container runtime supports running the image directly from shared storage, you may not strictly need this step—but staging to local NVMe tends to be faster and more predictable at large scale.

Because of the pre-validation, you can include this step in every job script without penalty—if the image is already present, it completes in milliseconds.

Getting Started

git clone https://github.com/edwardsp/mpi-stage.git cd mpi-stage make

For detailed usage and options, see the README.

Summary

mpi-stage started as a solution to a very specific problem—staging large container images efficiently across a large GPU cluster—but the same pattern may be useful in other scenarios where many nodes need the same large file.

By using MPI broadcasts, only a single node reads from the source filesystem, while data is distributed over the backend network using optimized collectives. In practice, this can significantly reduce load on shared filesystems and cloud-backed mounts, such as Azure Blob Storage accessed via blobfuse2, where hundreds of concurrent readers can otherwise become a bottleneck.

While container images were the initial focus, this approach could also be applied to staging training datasets, distributing model checkpoints or pretrained weights, or copying large binaries to local NVMe before a job starts. Anywhere that a “many nodes, same file” pattern exists is a potential fit.

If you're running large-scale containerized workloads on Azure HPC infrastructure, give it a try. If you use mpi-stage in other workflows, I'd love to hear what worked (and what didn't). Feedback and contributions are welcome.

Have questions or feedback? Leave a comment below or open an issue on GitHub.

Azure V710 V5 Series -AMD Radeon GPU - Validation of Siemens CAD -NX

Sunita_AZ0708 — Wed, 07 Jan 2026 16:38:08 GMT

Overview of Siemens NX

Siemens NX is a next-generation integrated CAD/CAM/CAE platform used by aerospace, automotive, industrial machinery, energy, medical, robotics, and defense manufacturers.
It spans:

Complex 3D modeling
Assemblies containing thousands to millions of parts
Surfacing and composites
Tolerance engineering
CAM and machining simulation
Integrated multi physics through Simcenter / NX Nastran

Because NX is used to design real-world engineered systems — aircraft structures, automotive platforms, satellites, robotic arms, injection molds — its usability and performance directly affect engineering velocity and product timelines.

NX Needs GPU Acceleration

NX is highly visual.
It leans heavily on:

OpenGL acceleration
Shader-based rendering
Hidden line removal
Real-time shading / material rendering
Ray-Traced Studio for photorealistic output

Switch shading modes → CAD content must stay readable
Zoom, section, annotate → requires stable frame pacing

NVads V710 v5-Series on Azure

The NVads V710 v5-series virtual machines on Azure are designed for GPU-accelerated workloads and virtual desktop environments. Key highlights:

Hardware Specs:

o GPU: AMD Radeon™ Pro V710 (up to 24 GiB frame buffer; fractional GPU options available).

o CPU: AMD EPYC™ 9V64 F (Genoa) with SMT, base frequency 3.95 GHz, peak 4.3 GHz.

o Memory: 16 GiB to 160 GiB.

o Storage: NVMe-based ephemeral local storage supported.

VM Sizes:

o Ranges from Standard_NV4ads_V710_v5 (4 vCPUs, 16 GiB RAM, 1/6 GPU) to Standard_NV28adms_V710_v5 (28 vCPUs, 160 GiB RAM, full GPU).

Supported Features:

o Premium storage, accelerated networking, ephemeral OS disk.

o Both Windows and Linux VMs supported.

o No additional GPU licensing is required.

AMD Radeon™ PRO GPUs offer:

o Optimized OpenGL professional driver stack

o Stable interactive performance vs large assemblies

Business Scenario Enabled by NX + Cloud GPU

Engineering Anywhere

Distributed teams can securely work on the same assemblies from any geographic region.

Supplier Ecosystem Collaboration

Tier-1/2 manufacturers and engineering partners can access controlled models without local high-end workstations.

Secure IP Protection

Data stays in Azure — files never leave the controlled workspace.

Faster Engineering Cycles

Visualization + simulation accelerate design reviews, decision making, and manufacturability evaluations.

Scalable Cost Model

Pay for compute only when needed — ideal for burst design cycles and testing workloads.

Architecture Overview – Siemens NX on Azure NVads_v710

Key Architecture Elements

- Create Azure Virtual Machine- NVads_v710_24
- Install Azure AMD V710 GPU drivers
- Deploy Azure File-based storage
  Hosting assemblies, metadata, drawing packages, PMI, simulation data.
- Configure Vnet with Accelerated Networking
- Install NX licenses and software.
- Install NXCP & ATS Test suites on the Virtual Machine

Qualitative Benchmark on Azure NVads_v710_24

Siemens has approved the following qualitative test results. The certification matrix update is currently in progress.

Technical variant:

Complex assemblies with thousands of components maintained smooth rotation, zooming, and selection, even under concurrent session load.

NXCP and ATS test results on NVads_v710_24

Non-Interactive test results:

Note: Execution Time (seconds)

ATS Non‑Interactive Test Results validate the correctness and stability of Siemens NX graphical rendering by comparing generated images against approved reference outputs. The minimal or zero pixel differences confirm deterministic and visually consistent rendering, indicating a stable GPU driver and visualization pipeline. The reported test execution times (in seconds) represent the duration required to complete each automated graphics validation scenario, demonstrating predictable and repeatable processing performance under non‑interactive conditions.

Interactive test results on Azure NVads_v710_24:

Note: Execution Time (seconds)

ATS Interactive Test Results evaluate Siemens NX graphics behavior during real‑time user interactions such as rotation, zoom, pan, sectioning, and view manipulation. The results demonstrate stable and consistent rendering during interactive workflows, confirming that the GPU driver and visualization stack reliably support user‑driven NX operations.
The measured execution times (in seconds) reflect the responsiveness of each interactive graphics operation, indicating predictable behavior under live, user‑controlled conditions rather than peak performance tuning.

NX CAD functions		Automatic Tests	Interactive Tests
Grace1 Basic Tests	GrPlayer_xp64.exe <FILE> Basic_Features.tgl	Passed!	Passed!
	GrPlayer_xp64.exe <FILE> Fog_Measurement_Clipping.tgl	Passed!	Passed!
	GrPlayer_xp64.exe <FILE> lighting.tgl	Passed!	Passed!
	GrPlayer_xp64.exe <FILE> Shadow_Bump_Environment.tgl	Passed!	Passed!
	GrPlayer_xp64.exe <FILE> Texture_Map.tgl	Passed!	Passed!
Grace2 Graphics Tests	GrPlayer_64.exe <FILE> GrACETrace.tgl	Passed!	Passed!
Grace2 Graphics Tests	GrPlayer_64.exe <FILE> GrACETrace.tgl	Passed!	Passed!

NXCP Test Scenarios
NXCP Test Scenarios		Automatic Tests
NXCP Gdat Tests	gdat_leg_xp64.exe -infile <FILE> leg_gfx_cert_1.cgi	Passed!
	gdat_leg_xp64.exe -infile <FILE> leg_gfx_cert_2.cgi	Passed!
	gdat_leg_xp64.exe -infile <FILE> leg_gfx_cert_4.cgi	Passed!
	gdat_leg_xp64.exe -infile <FILE> leg_gfx_cert_5.cgi	Passed!
	gdat_leg_xp64.exe -infile <FILE> leg_gfx_cert_6.cgi	Passed!
	gdat_leg_xp64.exe -infile <FILE> leg_gfx_cert_7.cgi	Passed!
	gdat_leg_xp64.exe -infile <FILE> leg_gfx_cert_8.cgi	Passed!
	gdat_leg_xp64.exe -infile <FILE> leg_gfx_cert_9.cgi	Passed!
	gdat_leg_xp64.exe -infile <FILE> leg_gfx_cert_10.cgi	Passed!
	gdat_leg_xp64.exe -infile <FILE> leg_gfx_cert_11.cgi	Passed!
	gdat_leg_xp64.exe -infile <FILE> leg_gfx_cert_12.cgi	Passed!
	gdat_leg_xp64.exe -infile <FILE> leg_gfx_cert_13.cgi	Passed!
	gdat_leg_xp64.exe -infile <FILE> leg_gfx_cert_14.cgi	Passed!
	gdat_leg_xp64.exe -infile <FILE> leg_gfx_cert_15.cgi	Passed!

Benefits Azure NVads_v710 (AMD GPU Platform for NX

Workstation-class AMD Radeon PRO graphics drivers baked into Azure
Ensures ISV-validated driver pipeline.
Excellent performance for CAD workloads
Makes GPU-accelerated NX accessible to wider user bases.
Remote engineering enablement
Critical for companies who now operate global design teams.
Elastic scale
Spin up GPU when development peaks; scale down when idle.

Conclusion:

Siemens NX on Azure NVads_v710 powered by AMD GPUs enables enterprise-class CAD/CAM/CAE experiences in the cloud. NX benefits directly from workstation-grade OpenGL optimization, shading stability, and Ray Traced Studio acceleration, allowing engineers to interact smoothly with large assemblies, run visualization workloads, and perform design reviews without local hardware dependencies.

Right‑sized GPU delivers workstation‑class experience at lower cost

The family enables fractional GPU allocation (down to 1/6 of a Radeon™ Pro V710), allowing Siemens NX deployments to be right‑sized per user role. This avoids over‑provisioning full GPUs while still delivering ISV‑grade OpenGL and visualization stability, resulting in a lower per‑engineer cost compared to fixed full‑GPU cloud or on‑prem workstations

Elastic scale improves cost efficiency for burst engineering workloads

NVads_V710_v5 instances support on demand scaling and ephemeral NVMe storage, allowing NX environments to scale up for design reviews, supplier collaboration, or peak integration cycles and scale down when idle. This consumption model provides a cost advantage over fixed on prem workstations that remain underutilized outside peak engineering periods

NX visualization pipelines benefit from balanced CPU–GPU architecture

The combination of high‑frequency AMD EPYC™ Genoa CPUs (up to 4.3 GHz) and Radeon™ Pro V710 GPUs addresses Siemens NX’s mixed CPU–GPU workload profile, where scene graph processing, tessellation, and OpenGL submission are CPU‑sensitive. This balance reduces idle GPU cycles, improving effective utilization and overall cost efficiency when compared with GPU‑heavy but CPU‑constrained configurations

The result is a scalable, secure, and cost-efficient engineering platform that supports distributed innovation, supplier collaboration, and digital product development workflows — all backed by the Rendering and interaction consistency of AMD GPU virtualization on Azure.