Tag:"azure kubernetes service" | Microsoft Community Hub

Azure Monitor Application Insights Auto-Instrumentation for Java and Node Microservices on AKS
Key Takeaways (TLDR) Monitor Java and Node applications with zero code changes Fast onboarding: just 2 steps Supports distributed tracing, logs, and metrics Correlates application-level telemetry in Application Insights with infrastructure-level telemetry in Container Insights Available today in public preview Introduction Monitoring your applications is now easier than ever with the public preview release of Auto-Instrumentation for Azure Kubernetes Service (AKS). You can now easily monitor your Java and Node deployments without changing your code by leveraging auto-instrumentation that is integrated into the AKS cluster. This feature is ideal for developers or operators who are... Looking to add monitoring in the easiest way possible, without modifying code and avoiding ongoing SDK update maintenance. Starting out on their monitoring journey and looking to benefit from carefully chosen default configurations with the ability to tweak them over time. Working with someone else’s code and looking to instrument at scale. Or considering monitoring for the first time at the time of deployment. Before the introduction of this feature, users needed to manually instrument code, install language-specific SDKs, and manage updates on their own—a process that involved significant effort and numerous opportunities for errors. Now, all you need to do is follow a simple two-step process to instrument your applications and automatically send correlated OpenTelemetry-based application-level logs, metrics, and distributed tracing to your Application Insights resource. With AKS Auto-Instrumentation, you will be able to assess the performance of your application and identify the cause of any incidents more efficiently using the robust application performance monitoring capabilities of Azure Monitor Application Insights. This streamlined approach not only saves time but also ensures that your monitoring setup is both reliable and scalable. Feature Enablement and Onboarding To onboard to this feature, you will need to follow a two-step process: Prepare your cluster by installing the application monitoring webhook. Choose between namespace-wide onboarding or per-deployment onboarding by creating K8’s custom resources. Namespace-wide onboarding is the easiest method. It allows you to instrument all Java or Node deployments in your namespace and direct telemetry to a single Application Insights resource. Per-deployment onboarding allows more control by targeting specific deployments and directing telemetry to different Application Insights resources. Once the custom resource is created, you will need to deploy or redeploy your application, and telemetry will start flowing to Application Insights. For step-by-step instructions and to learn more about onboarding visit our official documentation on MS Learn. The Application Insights experience Once telemetry begins flowing, you can take advantage of Application Insights features such as Application Map, Failures/Performance Views, Availability, and more to help you efficiently diagnose and troubleshoot application issues. Let’s look at an example: I have an auto-instrumented distributed application running in the demoapp namespace of my AKS cluster. It consists of: One Java microservice Two Node.js microservices MongoDB and Redis as its data layer Scenario: End users have been complaining about some latency in the application. As the DRI, I can start my troubleshooting journey by going to the Application Map to get a topological view of my distributed application. I open Application Map and notice MicroserviceA has a red border - 50% of calls are erroring. The Container Insights card shows healthy pods - no failed pods or high CPU/memory usage. I can eliminate infrastructure issues as the cause of the slowness. In the Performance card, I spot that the rescuepet operation has an average duration of 10 seconds. That's pretty long. I drill in to get a distributed trace of the operation and find the root cause: an OutOfMemoryError. In this scenario, the issue has been identified as an out-of-memory error at the application layer. However, when the root cause is not in the code but in the infrastructure I get a full set of resource properties with every distributed trace so I can easily identify the infra resources running each span of my trace. I can click the investigate pods button to transition to Azure Monitor Container Insights and investigate my pods further. This correlation between application-level and infrastructure-level telemetry makes it much easier to determine whether the issue is caused by the application or the infrastructure. Pricing There is no additional cost to use AKS auto-instrumentation to send data to Azure Monitor. You will be only charged as per the current pricing. What’s Next Language Support This integration supports Java and Node workloads by leveraging the Azure Monitor OpenTelemetry distro. We have distros for .NET and Python as well and we are working to integrate these distros into this solution. At that point, this integration will support .NET, Python, Java and Node.js. For customers that want to instrument workloads in other languages such as Go, Ruby, PHP, etc. we plan to leverage open-source instrumentations available in the Open Telemetry community. In this scenario, customers will instrument their code using open source OpenTelemetry instrumentations, and we will provide mechanisms that will make it easy to channel the telemetry to Application Insights. Application Insights will expose an endpoint that accepts OpenTelemetry Language Protocol (OTLP) signals and configure the instrumented workload to channel the telemetry to this endpoint. Operating Systems and K8’s Controllers Right now, you can only instrument kubernetes deployments running on Linux node pools, but we plan to expand support to introduce support for Linux ARM64 node pools as well as support for StatefulSet, Job, Cronjob, and Replicaset controller types. Portal Experiences We are also working on Azure portal experiences to make onboarding easier. When our portal experiences for onboarding are released, users will be able to install the Application Insights extension for AKS using the portal and use a portal user interface to instrument their workloads instead of having to create custom resources. Beyond onboarding, we are working to build Application Insights consumption experiences within the AKS namespace and workloads blade. You will be able to see application-level telemetry right there in the AKS portal without having to navigate away from your cluster to Application Insights. FAQs: What are the advantages of AKS Auto-Instrumentation? No code changes required No access to source code required No configuration changes required Eliminates instrumentation maintenance What languages are supported by AKS Auto-Instrumentation? Currently, AKS Auto-Instrumentation supports Java and Node.js applications. Python and .NET support is coming soon. Moreover, we will be adding support for all OTel supported languages like Go soon via native OTLP ingestion. Does AKS Auto-Instrumentation support custom metrics? For Node.js applications, custom metrics require manual instrumentation with the Azure Monitor OpenTelemetry Distro. Java applications allow custom metrics with auto-instrumentation. Click here for more FAQs. This article was co-authored by Rishab Jolly and Abinet Abate
abinetabate
Apr 16, 2025 Place Azure Observability Blog
228Views
0likes
0Comments
Get Ready for .NET Conf: Focus on Modernization
We’re excited to announce the topics and speakers for .NET Conf: Focus on Modernization, our latest virtual event on April 22-23, 2025! This event features live sessions from .NET and cloud computing experts, providing attendees with the latest insights into modernizing .NET applications, including technical upgrades, cloud migration, and tooling advancements. To get ready, visit the .NET Conf: Focus on Modernization home page and click Add to Calendar so you can save the date on your calendar. From this page, on the day of the event you’ll be able to join a live stream on YouTube and Twitch. We will also make the source code for the demos available on GitHub and the on-demand replays will be available on our YouTube channel. Learn more: https://focus.dotnetconf.net/ Why attend? In the fast-changing technological environment we now find ourselves, it has never been more urgent to modernize enterprise .NET applications to maintain competitiveness and stay ahead of the next innovation. Updating .NET applications for the cloud is a major business priority and involves not only technical upgrades and cloud migration, but also improvements in tooling, processes, and skills. At this event, you will get the end to end insights across latest tools, innovations, and best practices for successful .NET modernization. What can developers expect? The event will run live for up to five hours each day, covering different aspects of .NET modernizations. Scott Hanselman will set the tone for day one with discussion of the experiences and processes to modernize .NET applications in the era of AI. This will be followed by expert sessions on upgrading .NET apps and modernizing both your apps and data to the cloud. Day two will soar higher into the clouds, with sessions to help with cloud migration, cloud development, and infusing AI into your apps. You can interact with experts and ask questions to deepen your expertise, as we broadcast live on YouTube, or Twitch. Recordings of all sessions will be available with materials after the event. Agenda Here’s a quick snapshot of the schedule. Things may change, and we recommend that you please visit the event home page for the latest agenda and session times: https://focus.dotnetconf.net/agenda Day 1 – April 22, Tuesday Time (PDT) Session 8:00 am Modernizing .NET: Future-ready applications in the era of AI Scott Hanselman, Chet Husk, McKenna Barlow 9:00 am Deep dive into the upcoming AI-assisted tooling to upgrade .NET apps Chet Husk, McKenna Barlow 10:00 am Use Reliable Web App patterns to confidently replatform your web apps Pablo Lopes 11:00 am Modernize Data-Driven Apps (No AI Needed) Jerry Nixon 12:00 pm Modernize from ASP.NET to ASP.NET Core: The Future is Now Taylor Southwick Day 2 – April 23, Wednesday Time (PDT) Session 8:00 am Unblock .NET modernization with AI-assisted app and code assessment tools Michael Yen-Chi Ho 9:00 am Cloud development doesn't have to be painful thanks to .NET Aspire Maddy Montaquila (Leger) 10:00 am Introducing Artificial Intelligence to your application Jordan Matthiesen 11:00 am Modernizing your desktop: From WinForms to Blazor, Azure, and AI Santiago Arango Toro Save the Date! .NET Conf: Focus on Modernization is a free, two-day livestream event that you won’t want to miss. Tune in on April 22 and 23, 2025, ask questions live, and learn how to get your .NET applications ready for the AI revolution. Save the date! Stay tuned for more updates and detailed session information. We can’t wait to see you there!
Mayunk_Jain
Apr 15, 2025 Place Apps on Azure Blog
638Views
0likes
0Comments
Code the Future with Java and AI – Join Me at JDConf 2025
JDConf 2025 is just around the corner, and whether you’re a Java developer, architect, team leader, or decision maker I hope you’ll join me as we explore how Java is evolving with the power of AI and how you can start building the next generation of intelligent applications today. Why JDConf 2025? With over 22 expert-led sessions and 10+ hours of live content, JDConf is packed with learning, hands-on demos, and real-world solutions. You’ll hear from Java leaders and engineers on everything from modern application design to bringing AI into your Java stack. It’s free, virtual and your chance to connect from wherever you are. (On-demand sessions will also be available globally from April 9–10, so you can tune in anytime from anywhere.) Bring AI into Java Apps At JDConf 2025, we are going beyond buzzwords. We’ll show you how to bring AI into real Java apps, using patterns and tools that work today. First, we’ll cover Retrieval-Augmented Generation (RAG), a design pattern where your app retrieves the right business data in real time, and combines it with AI models to generate smart, context-aware responses. Whether it is answering support queries, optimizing schedules, or generating insights, RAG enables your app to think in real time. Second, we’ll introduce AI agents -- software entities that do more than respond. They act. Think about automating production line scheduling at an auto manufacturer or rebooking delayed flights for passengers. These agents interact with APIs, reason over data, and make decisions, all without human intervention. Third, we’ll explore the complete AI application platform on Azure. It is built to work with the tools Java developers already know - from Spring Boot to Quarkus - and includes OpenAI and many other models, vector search with PostgreSQL, and libraries like Spring AI and LangChain4j. Here are just two example stacks: Spring Boot AI Stack: any app hosting services like Azure Container Apps or App Service + Spring AI + OpenAI + PostgreSQL for business data and vector data store. Quarkus AI Stack: any app hosting services like Azure Container Apps or App Service + LangChain4j + OpenAI + PostgreSQL for business data and vector data store. This is how you turn existing Java apps into intelligent, interactive systems, without reinventing everything. Whether you are an experienced developer or just starting out, JDConf offers valuable opportunities to explore the latest advancements in Java, cloud, and AI technologies; gain practical insights; and connect with Java experts from across the globe – including Java 25, Virtual Threads, Spring Boot, Jakarta EE 12, AI developer experiences, Spring AI, LangChain4j, combining data and AI, automated refactoring to Java app code modernization. We’ll also show you how GitHub Copilot helps you modernize faster. GitHub Copilot's new “upgrade assistant” can help refactor your project, suggest dependency upgrades, and guide you through framework transitions, freeing you up to focus on innovation. Get the Right Fit for Your Java App And what if your apps run on JBoss, WebLogic, or Tomcat? We will walk you through how to map those apps to the right Azure service: Monoliths (JAR, WAR, EAR) → Deploy to App Service Microservices or containers → Use Azure Container Apps or AKS WebLogic & WebSphere → Lift and shift to Azure Virtual Machines JBoss EAP containers → Run on Azure Red Hat OpenShift You’ll get clear guidance on where your apps fit and how to move forward, with no guesswork or dead ends. Let's Code the Future, Together I’ll be there, along with Josh Long from the Spring AI community and Lize Raes from the LangChain4j community, delivering a technical keynote packed with practical insights. If you haven’t started building intelligent Java apps, you can start with JDConf. If you’ve already started on the journey, tune in to learn how you can enrich your experiences with the latest in tech. So, mark your calendar. Spread the word. Bring your team. JDConf 2025 is your place to build what is next with Java and AI. 👉 Register now at jdconf.com. Check out the 20+ exclusive sessions brought to you by Java experts from across the globe in all major time zones.
AmandaKSilver
Apr 03, 2025 Place Apps on Azure Blog
143Views
0likes
0Comments
Supercharge Azure Files performance for metadata-intensive workloads
Handling millions—or billions—of small files is business as usual for many cloud workloads. But behind the scenes, it’s not just about reading and writing data—it's the constant file opens, closes, directory listings, and existence checks that really impact performance. These metadata operations may seem small, but they’re critical—and can become a major bottleneck if they’re not fast. From AI/ML workloads on Azure Kubernetes Service, to web apps like Moodle, to CI/CD pipelines and virtual desktops—many applications are metadata intensive. And when every millisecond counts, latency in these operations can drag down the entire experience. That’s why we’re excited to introduce a major boost in metadata performance. Applications experience up to 55% lower latency and 2–3x more consistent response times, ensuring greater reliability. Workloads with high metadata interaction, such as AI/ML pipelines, see the biggest gains with 3x higher parallel metadata IOPS for improved efficiency and scalability. Removing metadata bottlenecks allows more data operations too. We've seen workloads increase data IOPS and throughput up to 60%. In Azure Files SSD (premium), this enhancement accelerates metadata operations for both SMB and REST protocols, benefiting new and existing file shares at no extra cost. Whether you're running critical business applications, scaling DevOps workflows, or supporting thousands of virtual desktop users, Azure Files is now faster, more scalable, and optimized for your most demanding workloads. Metadata Caching accelerates real-world solutions. GIS on Azure Virtual Desktop GIS (Geographic Information System) workloads are crucial for analyzing and managing spatial data, supporting industries like urban planning, agriculture, and disaster management. By visualizing spatial relationships, GIS helps organizations make better decisions about infrastructure and resource management. Azure Virtual Desktop (AVD) is a popular choice for hosting GIS workloads in the cloud. These workloads often experience performance bottlenecks due to frequent interactions with large volumes of smaller files on shared file storage. Metadata caching reduces latency and accelerates file interactions—such as opening and closing these files—enabling faster data access and improving GIS job execution in virtual desktop environments. Customers, like Suncor Energy, are already experiencing the impact of Metadata Caching in GIS workloads. “Enabling Metadata Cache in Azure Files SSD (premium) significantly improved geospatial (GIS) workload performance, reducing execution time by 43.18%. This enhancement boosts throughput and IOPS, increasing the value of Azure Files.” — Colin Stuckless, Suncor Energy Inc. Moodle Web Services Moodle is a comprehensive learning management system (LMS) that combines server hosting, databases (such as MySQL or PostgreSQL), file storage (using Azure Files SSD), and PHP-based web servers. It’s designed to facilitate course management, allowing instructors to upload materials, assignments, and quizzes. Moodle requires frequent read/write requests for course materials, assignments, and user interactions generating a high volume of metadata lookups, particularly when accessing shared content or navigating large course repositories. With Metadata Caching, Moodle operates faster and more efficiently. Response times have improved by 33%, reducing wait times for students and instructors when accessing materials or submitting work. These enhancements also boost Moodle’s scalability, enabling it to support 3x more students and user sessions without compromising performance. Even during peak usage, when many users are active simultaneously, Moodle remains stable and responsive. As a result, students can access resources and submit work more quickly, while instructors can manage larger courses and assignments more effectively. GitHub Actions on Azure Kubernetes Service (AKS) GitHub Actions is a powerful automation tool seamlessly integrated with GitHub, enabling developers to build, test, and deploy code directly from their repositories. By leveraging Azure Kubernetes Service (AKS), GitHub Actions automates tasks through workflows defined in AKS YAML files, facilitating efficient container orchestration, scaling, and deployment of applications within a Kubernetes environment. These workflows can be triggered by various events, such as code pushes, pull requests, or even scheduled times, streamlining the development process and enhancing efficiency. These operations generate a high volume of metadata lookups, as each workflow execution involves checking for updated dependencies, accessing cached libraries, and continuously writing execution logs. Metadata caching significantly reduces the time required to retrieve and process metadata, resulting in quicker build artifact handling and smoother, more efficient deployment cycles. As a result, pipeline execution is 57% faster, allowing developers to build and deploy in half the time! How to get started You can now supercharge your Azure Files performance by enabling metadata caching for your applications today, at no extra cost! So don’t wait! To get started, register your subscription with the Metadata Cache feature using Azure portal or PowerShell to enable all new and existing accounts with Metadata Caching. Metadata Cache is now generally available in multiple regions, with more being added as we expand coverage. For Regional Availability please visit the following Link
agroves54
Apr 02, 2025 Place Azure Storage Blog
539Views
0likes
0Comments
Deploying Azure ND H100 v5 Instances in AKS with NVIDIA MIG GPU Slicing
In this article we will cover: AKS Cluster Deployment (Latest Version) – creating an AKS cluster using the latest Kubernetes version. GPU Node Pool Provisioning – adding an ND H100 v5 node pool on Ubuntu, with --skip-gpu-driver-install to disable automatic driver installation. NVIDIA H100 MIG Slicing Configurations – available MIG partition profiles on the H100 GPU and how to enable them. Workload Recommendations for MIG Profiles – choosing optimal MIG slice sizes for different AI/ML and HPC scenarios. Best Practices for MIG Management and Scheduling – managing MIG in AKS, scheduling pods, and operational tips. AKS Cluster Deployment (Using the Latest Version) Install/Update Azure CLI: Ensure you have Azure CLI 2.0.64+ (or Azure CLI 1.0.0b2 for preview features). This is required for using the --skip-gpu-driver-install option and other latest features. Install the AKS preview extension if needed: az extension add --name aks-preview az extension update --name aks-preview (Preview features are opt-in; using the preview extension gives access to the latest AKS capabilities) Create a Resource Group: If not already done, create an Azure resource group for the cluster: az group create -n MyResourceGroup -l eastus Create the AKS Cluster: Run az aks create to create the AKS control plane. You can start with a default system node pool (e.g. a small VM for system pods) and no GPU nodes yet. For example: az aks create -g MyResourceGroup -n MyAKSCluster \ --node-vm-size Standard_D4s_v5 \ --node-count 1 \ --kubernetes-version <latest-stable-version> \ --enable-addons monitoring This creates a cluster named MyAKSCluster with one standard node. Use the --kubernetes-version flag to specify the latest AKS-supported Kubernetes version (or omit it to get the default latest). As of early 2025, AKS supports Kubernetes 1.27+; using the newest version ensures support for features like MIG and the ND H100 v5 SKU. Retrieve Cluster Credentials: Once created, get your Kubernetes credentials: az aks get-credentials -g MyResourceGroup -n MyAKSCluster Verification: After creation, you should have a running AKS cluster. You can verify the control plane is up with: kubectl get nodes Adding an ND H100 v5 GPU Node Pool (Ubuntu + Skip Driver Install) Next, add a GPU node pool using the ND H100 v5 VM size. The ND H100 v5 series VMs each come with 8× NVIDIA H100 80GB GPUs (640 GB total GPU memory), high-bandwidth interconnects, and 96 vCPUs– ideal for large-scale AI and HPC workloads. We will configure this node pool to run Ubuntu and skip the automatic NVIDIA driver installation, since we plan to manage drivers (and MIG settings) manually or via the NVIDIA operator. Steps to add the GPU node pool: Use Ubuntu Node Image: AKS supports Ubuntu 20.04/22.04 for ND H100 v5 nodes. The default AKS Linux OS (Ubuntu) is suitable. We also set --os-sku Ubuntu to ensure we use Ubuntu (if your cluster’s default is Azure Linux, note that Azure Linux is not currently supported for MIG node pools). Add the GPU Node Pool with Azure CLI: Run: az aks nodepool add \ --cluster-name MyAKSCluster \ --resource-group MyResourceGroup \ --name h100np \ --node-vm-size Standard_ND96isr_H100_v5 \ --node-count 1 \ --node-os-type Linux \ --os-sku Ubuntu \ --skip-gpu-driver-install \ --node-taints nvidia.com/gpu=true:NoSchedule Let’s break down these parameters: --node-vm-size Standard_ND96isr_H100_v5 selects the ND H100 v5 VM size (96 vCPUs, 8×H100 GPUs). Ensure your subscription has quota for this SKU and region. --node-count 1 starts with one GPU VM (scale as needed). --skip-gpu-driver-install (Preview) tells AKS not to pre-install NVIDIA drivers on the node. This prevents the default driver installation, because we plan to handle drivers ourselves (using NVIDIA’s GPU Operator for better control). When using this flag, new GPU nodes come up without NVIDIA drivers until you install them manually or via an operator--node-taints --node-taints nvidia.com/gpu=true:NoSchedule taints the GPU nodes so that regular pods won’t be scheduled on them accidentally. Only pods with a matching toleration (e.g. labeled for GPU use) can run on these nodes. This is a best practice to reserve expensive GPU nodes for GPU workloads (Optional) You can also add labels if needed. For example, to prepare for MIG configuration with the NVIDIA operator, you might add a label like nvidia.com/mig.config=all-1g.10gb to indicate the desired MIG slicing (explained later). We will address MIG config shortly, so adding such a label now is optional. Wait for Node Pool to be Ready: Monitor the Azure CLI output or use kubectl get nodes until the new node appears. It should register in Kubernetes (in NotReady state initially while it's configuring). Since we skipped driver install, the node will not have GPU scheduling resources yet (no nvidia.com/gpu resource visible) until we complete the next step. Installing the NVIDIA Driver Manually (or via GPU Operator) Because we used --skip-gpu-driver-install, the node will not have the necessary NVIDIA driver or CUDA runtime out of the box. You have two main approaches to install the driver: Use the NVIDIA GPU Operator (Helm-based) to handle driver installation. Install drivers manually (e.g., run a DaemonSet that downloads and installs the .run package or Debian packages). NVIDIA GPU Operator manages drivers, the Kubernetes device plugin, and GPU monitoring components. AKS GPU node pools come with the NVIDIA drivers and container runtime already pre-installed. BUT, because we used the flag : -skip-gpu-driver-install, we can now deploy the NVIDIA GPU Operator to handle GPU workloads and monitoring, while disabling its driver installation (to avoid conflicts with the pre-installed drivers). The GPU Operator will deploy the necessary components like the Kubernetes device plugin and the DCGM exporter for monitoring. 2.1 Installing via NVIDIA GPU Operator Step 1: Add the NVIDIA Helm repository. NVIDIA provides a Helm chart for the GPU Operator. Add the official NVIDIA Helm repo and update it: helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update This repository contains the gpu-operator chart and other NVIDIA helm charts Step 2: Install the GPU Operator via Helm. Use Helm to install the GPU Operator into a dedicated namespace (e.g., gpu-operator). In AKS, disable the GPU Operator’s driver and toolkit deployment (since AKS already has those), and specify the correct container runtime class for NVIDIA. For example: helm install gpu-operator nvidia/gpu-operator \ -n gpu-operator --create-namespace \ --set operator.runtimeClass=nvidia-container-runtime In the above command: operator.runtimeClass=nvidia-container-runtime aligns with the runtime class name configured on AKS for GPU support After a few minutes, Helm should report a successful deployment. For example: NAME: gpu-operator LAST DEPLOYED: Fri May 5 15:30:05 2023 NAMESPACE: gpu-operator STATUS: deployed REVISION: 1 TEST SUITE: None You can verify that the GPU Operator’s pods are running in the cluster. The Operator will deploy several DaemonSets including the NVIDIA device plugin, DCGM exporter, and others. For example, after installation you should see pods like the following in the gpu-operator namespace: nvidia-dcgm-exporter-xxxxx 1/1 Running 0 60s nvidia-device-plugin-daemonset-xxxxx 1/1 Running 0 60s nvidia-mig-manager-xxxxx 1/1 Running 0 4m nvidia-driver-daemonset-xxxxx 1/1 Running 0 4m gpu-operator-node-feature-discovery-... 1/1 Running 0 5m ... (other GPU operator pods) ... Here we see the NVIDIA device plugin and NVIDIA DCGM exporter pods running on each GPU node, as well as other components. (Note: In our AKS setup, the nvidia-driver-daemonset may be present but left idle since we disabled driver management.) Step 3: Confirm the operator’s GPU validation. The GPU Operator will run a CUDA validation job to verify everything is working. Check that the CUDA validation pod has completed successfully: kubectl get pods -n gpu-operator -l app=nvidia-cuda-validator Expected output: NAME READY STATUS RESTARTS AGE nvidia-cuda-validator-bpvkt 0/1 Completed 0 3m56s A Completed CUDA validator indicates the GPUs are accessible and the NVIDIA stack is functioning. At this point, you have the NVIDIA GPU Operator (with device plugin and DCGM exporter) installed via Helm on AKS. Verifying MIG on H100 with Node Pool Provisioning Once the driver is installed and the NVIDIA device plugin is running, you can verify MIG. The process is similar to verifying MIG on A100, but the resource naming and GPU partitioning reflect H100 capabilities. Check Node Resources kubectl describe node <h100-node-name> If you chose single MIG strategy, you might see: Allocatable: nvidia.com/gpu: 56 for a node with 8 H100s × 7 MIG slices each = 56. Or: nvidia.com/gpu: 14 if you used MIG2g (which yields 2–3 slices per GPU, depending on the exact profile). If you chose mixed MIG strategy (mig.strategy=mixed), you’ll see something like: Allocatable: nvidia.com/mig-1g.10gb: 56 or whichever MIG slice name is appropriate (e.g., mig-3g.40gb for MIG3g). Confirm MIG in nvidia-smi Run a GPU Workload For instance, run a quick CUDA container: kubectl run mig-test --rm -ti \ --image=nvidia/cuda:12.1.1-runtime-ubuntu22.04 \ --limits="nvidia.com/gpu=1" \ -- bash Inside the container, nvidia-smi should confirm you have a MIG device. Then any CUDA commands (e.g., deviceQuery) should pass, indicating MIG is active and the driver is working. nvidia-smi -L MIG Management on H100 The H100 supports several MIG profiles – predefined ways to slice the GPU. Each profile is denoted by <N>g.<M>gb meaning it uses N GPU compute slices (out of 7) and M GB of memory. Key H100 80GB MIG profiles include: MIG 1g.10gb: Each instance has 1/7 of the SMs and 10 GB memory (1/8 of VRAM). This yields 7 instances per GPU (7 × 10 GB = 70 GB out of 80, a small portion is reserved). This is the smallest slice size and maximizes the number of instances (useful for many lightweight tasks). MIG 1g.20gb: Each instance has 1/7 of SMs but 20 GB memory (1/4 of VRAM), allowing up to 4 instances per GPU. This profile gives each instance more memory while still only a single compute slice – useful for memory-intensive workloads that don’t need much compute. MIG 2g.20gb: Each instance gets 2/7 of SMs and 20 GB memory (2/8 of VRAM). 3 instances can run on one GPU. This offers a balance: more compute per instance than 1g, with a moderate 20 GB memory each. MIG 3g.40gb: Each instance has 3/7 of SMs and 40 GB memory (half the VRAM). Two instances fit on one H100. This effectively splits the GPU in half. MIG 4g.40gb: Each instance uses 4/7 of SMs and 40 GB memory. Only one such instance can exist per GPU (because it uses half the memory and more than half of the SMs). In practice, a 4g.40gb profile might be combined with a smaller profile on the same GPU (e.g., a 4g.40gb + a 3g.40gb could occupy one GPU, totaling 7/7 SM and 80GB). However, AKS node pools use a single uniform profile per GPU, so you typically wouldn’t mix profiles on the same GPU in AKS. MIG 7g.80gb: This profile uses the entire GPU (all 7/7 SMs and 80 GB memory). Essentially, MIG 7g.80gb is the full GPU as one instance (no slicing). It’s equivalent to not using MIG at all for that GPU. These profiles illustrate the flexibility: you can trade off number of instances vs. the power of each instance. For example, MIG 1g.10gb gives you seven small GPUs, whereas MIG 3g.40gb gives you two much larger slices (each roughly half of an H100). All MIG instances are hardware-isolated, meaning each instance’s performance is independent (one instance can’t starve others of GPU resources) Enabling MIG in AKS: There are two main ways to configure MIG on the AKS node pool: At Node Pool Creation (Static MIG Profile): Azure allows specifying a GPU instance profile when creating the node pool. For example, adding --gpu-instance-profile MIG1g to the az aks nodepool add command would provision each H100 GPU in 1g mode (e.g., 7×10GB instances per GPU). Supported profile names for H100 include MIG1g, MIG2g, MIG3g, MIG4g, and MIG7g (the same profile names used for A100, but on H100 they correspond to the sizes above). Important: Once set, the MIG profile on a node pool cannot be changed without recreating the node pool. If you chose MIG1g, all GPUs in that node pool will be partitioned into 7 slices each, and you can’t later switch those nodes to a different profile on the fly. Dynamically via NVIDIA GPU Operator: If you skipped the driver install (as we did) and are using the GPU Operator, you can let the operator manage MIG. This involves labeling the node with a desired MIG layout. For example, nvidia.com/mig.config=all-1g.10gb means “partition all GPUs into 1g.10gb slices.” The operator’s MIG Manager will then enable MIG mode on the GPUs, create the specified MIG instances, and mark the node ready when done. This approach offers flexibility – you could theoretically adjust the MIG profile by changing the label and letting the operator reconfigure (though it will drain and reboot the node to apply changes). The operator adds a taint like mig-nvidia.io/device-config=pending (or similar) during reconfiguration to prevent scheduling pods too early For our deployment, we opted to skip Azure’s automatic MIG config and use the NVIDIA operator. If you followed the steps in section 2 and set the nvidia.com/mig.config label before node creation, the node on first boot will come up, install drivers, then partition into the specified MIG profile. If not, you can label the node now and the operator will configure MIG accordingly. For example: kubectl label node <node-name> nvidia.com/mig.config=all-3g.40gb --overwrite to split each GPU into two 3g.40gb instances. The operator will detect this and partition the GPUs (the node may briefly go NotReady while MIG is being set up). After MIG is configured, verify the node’s GPU resources again. Depending on the MIG strategy (see next section), you will either see a larger number of generic nvidia.com/gpu resources or specifically named resources like nvidia.com/mig-3g.40gb. We will discuss how to schedule workloads onto these MIG instances next. Important Considerations: Workload Interruption: Applying a new MIG configuration can disrupt running GPU workloads. It's advisable to drain the node or ensure that no critical workloads are running during the reconfiguration process. Node Reboot: Depending on the environment and GPU model, enabling or modifying MIG configurations might require a node reboot. Ensure that your system is prepared for potential reboots to prevent unexpected downtime. Workload Recommendations for MIG Profiles (AI/ML vs. HPC) Different MIG slicing configurations are suited to different types of workloads. Here are recommendations for AI/ML and HPC scenarios: Full GPU (MIG 7g.80gb or MIG disabled) – Best for the largest and most intensive tasks. If you are training large deep learning models (e.g. GPT-style models, complex computer vision training) or running HPC simulations that fully utilize a GPU, you should use the entire H100 GPU. The ND H100 v5 is designed to excel at these demanding workloads. In Kubernetes, you would simply schedule pods that request a whole GPU. (If MIG mode is enabled with 7g.80gb profile, each GPU is one resource unit.) This ensures maximum performance for jobs that can utilize 80 GB of GPU memory and all compute units. HPC workloads like physics simulations, CFD, weather modeling, etc., typically fall here – they are optimized to use full GPUs or even multiple GPUs in parallel, so slicing a GPU could impede their performance unless you explicitly want to run multiple smaller HPC jobs on one card. Large MIG Partitions (3g.40gb or 4g.40gb) – Good for moderately large models or jobs that don’t quite need a full H100. For instance, you can split an H100 into 2× 3g.40gb instances, each with 40 GB VRAM and ~43% of the H100’s compute. This configuration is popular for AI model serving and inference where a full H100 might be underutilized. In fact, it might happen that two MIG 3g.40gb instances on an H100 can serve models with performance equal or better than two full A100 GPUs, at a lower cost. Each 3g.40gb slice is roughly equivalent to an A100 40GB in capability, and also unlocks H100-specific features (like FP8 precision for inference). Use cases: Serving two large ML models concurrently (each model up to 40GB in size, such as certain GPT-XXL or vision models). Each model gets a dedicated MIG slice. Running two medium-sized training jobs on one physical GPU. For example, two separate experiments that each need ~40GB GPU memory can run in parallel, each on a MIG 3g.40gb. This can increase throughput for hyperparameter tuning or multi-user environments. HPC batch jobs: if you have HPC tasks that can fit in half a GPU (perhaps memory-bound tasks or jobs that only need ~50% of the GPU’s FLOPs), using two 3g.40gb instances allows two jobs to run on one GPU server concurrently with minimal interference. MIG 4g.40gb (one 40GB instance using ~57% of compute) is a less common choice by itself – since only one 4g instance can exist per GPU, it leaves some GPU capacity unused (the remaining 3/7 SMs would be idle). It might be used in a mixed profile scenario (one 4g + one 3g on the same GPU) if manually configured. In AKS (which uses uniform profiles per node pool), you’d typically prefer 3g.40gb if you want two equal halves, or just use full GPUs. So in practice, stick with 3g.40gb for a clean 2-way split on H100. Medium MIG Partitions (2g.20gb) – Good for multiple medium workloads. This profile yields 3 instances per GPU, each with 20 GB memory and about 28.6% of the compute. This is useful when you have several smaller training jobs or medium-sized inference tasks that run concurrently. Examples: Serving three different ML models (each ~15–20 GB in size) from one H100 node, each model on its own MIG 2g.20gb instance. Running 3 parallel training jobs for smaller models or prototyping (each job can use 20GB GPU memory). For instance, three data scientists can share one H100 GPU server, each getting what is effectively a “20GB GPU”. Each 2g.20gb MIG slice should outperform a V100 (16 GB) in both memory and compute, so this is still a hefty slice for many models. In HPC context, if you had many lighter GPU-accelerated tasks (for example, three independent tasks that each use ~1/3 of a GPU), this profile could allow them to share a node efficiently. Small MIG Partitions (1g.10gb) – Ideal for high-density inference and lightweight workloads. This profile creates 7 instances per GPU, each with 10 GB VRAM and 1/7 of the compute. It’s perfect for AI inference microservices, model ensembles, or multi-tenant GPU environments: Deploying many small models or instances of a model. For example, you could host seven different AI services (each requiring <10GB GPU memory) on one physical H100, each in its own isolated MIG slice. Most cloud providers use this to offer “fractional GPUs” to customers– e.g., a user could rent a 1g.10gb slice instead of the whole GPU. Running interactive workloads like Jupyter notebooks or development environments for multiple users on one GPU server. Each user can be assigned a MIG 1g.10gb slice for testing small-scale models or doing data science workloads, without affecting others. Inference tasks that are memory-light but require GPU acceleration – e.g., running many inference requests in parallel across MIG slices (each slice still has ample compute for model scoring tasks, and 10 GB is enough for many models like smaller CNNs or transformers). Keep in mind that 1g.10gb slices have the lowest compute per instance, so they are suited for workloads that individually don’t need the full throughput of an H100. They shine when throughput is achieved by running many in parallel. 1g.20gb profile – This one is a bit niche: 4 slices per GPU, each with 20 GB but only 1/7 of the SMs. You might use this if each task needs a large model (20 GB) but isn’t compute-intensive. An example could be running four instances of a large language model in inference mode, where each instance is constrained by memory (loading a 15-18GB model) but you deliberately limit its compute share to run more concurrently. In practice, the 2g.20gb profile (which gives the same memory per instance and more compute) might be preferable if you can utilize the extra SMs. So 1g.20gb would only make sense if you truly have compute-light, memory-heavy workloads or if you need exactly four isolated instances on one GPU. HPC Workloads Consideration: Traditional HPC jobs (MPI applications, scientific computing) typically either use an entire GPU or none. MIG can be useful in HPC for capacity planning – e.g., running multiple smaller GPU-accelerated jobs simultaneously if they don’t all require a full H100. But it introduces complexity, as the HPC scheduler must be aware of fractional GPUs. Many HPC scenarios might instead use whole GPUs per job for simplicity. That said, for HPC inference or analytics (like running multiple inference tasks on simulation output), MIG slicing can improve utilization. If jobs are latency-sensitive, MIG’s isolation ensures one job doesn’t impact another, which is beneficial for multi-tenant HPC clusters (for example, different teams sharing a GPU node). In summary, choose the smallest MIG slice that still meets your workload’s requirements. This maximizes overall GPU utilization and cost-efficiency by packing more tasks on the hardware. Use larger slices or full GPUs only when a job truly needs the extra memory and compute. It’s often a good strategy to create multiple GPU node pools with different MIG profiles tailored to different workload types (e.g., one pool of full GPUs for training and one pool of 1g or 2g MIG GPUs for inference). Appendix A: MIG Management via AKS Node Pool Provisioning (without GPU Operator MIG profiles) Multi-Instance GPU (MIG) allows partitioning an NVIDIA A100 (and newer) GPU into multiple instances. AKS supports MIG for compatible GPU VM sizes (such as the ND A100 v4 series), but MIG must be configured when provisioning the node pool – it cannot be changed on the fly in AKS. In this section, we show how to create a MIG-enabled node pool and integrate it with Kubernetes scheduling. We will not use the GPU Operator’s dynamic MIG reconfiguration; instead, we set MIG at node pool creation time (which is the only option on AKS). Step 1: Provision an AKS node pool with a MIG profile. Choose a MIG-capable VM size (for example, Each instance has 1/7 of the SMs and 10 GB memory (1/8 of VRAM). This yields 7 instances per GPU (7 × 10 GB = 70 GB out of 80, a small portion is reserved). Use the Azure CLI to create a new node pool and specify the --gpu-instance-profile: az aks nodepool add \ --resource-group <myResourceGroup> \ --cluster-name <myAKSCluster> \ --name migpool \ --node-vm-size Standard_ND96isr_H100_v5 \\ --node-count 1 \ --gpu-instance-profile MIG1g In this example, we create a node pool named "migpool" with MIG profile MIG1g (each physical H100 GPU is split into 7 instances of 1g/5gb each). Important: You cannot change the MIG profile after the node pool is created. If you need a different MIG configuration (e.g., 2g.10gb or 4g.20gb instances), you must create a new node pool with the desired profile. Note: MIG is only supported on Ubuntu-based AKS node pools (not on Azure Linux nodes), and currently the AKS cluster autoscaler does not support scaling MIG-enabled node pools. Plan capacity accordingly since MIG node pools can’t auto-scale. Appendix B: Key Points and Best Practices No On-the-Fly Profile Changes With AKS, once a node pool is created with --gpu-instance-profile MIGxg, you cannot switch to a different MIG layout on that same node pool. If you need a new MIG profile, create a new node pool. --skip-gpu-driver-install This is typically used if you need a specific driver version, or if you want the GPU Operator to manage drivers (instead of the in-box AKS driver). Make sure your driver is installed before you schedule GPU workloads. If the driver is missing, pods that request GPU resources will fail to initialize. Driver Versions for H100 H100 requires driver branch R525 or newer (and CUDA 12+). Verify the GPU Operator or your manual install uses a driver that supports H100 and MIG on H100 specifically. Single vs. Mixed Strategy Single strategy lumps all MIG slices together as nvidia.com/gpu. This is simpler for uniform MIG node pools. Mixed strategy exposes resources like nvidia.com/mig-1g.10gb. Use if you need explicit scheduling by MIG slice type. Configure this in the GPU Operator’s Helm values (e.g., --set mig.strategy=single or mixed). If the Operator’s MIG Manager is disabled, it won’t attempt to reconfigure MIG, but it will still let the device plugin report the slices in single or mixed mode. Resource Requests and Scheduling If using single strategy, a pod that requests nvidia.com/gpu: 1 will be allocated a single 1g.10gb MIG slice on H100. If using mixed, that same request must specifically match the MIG resource name (e.g., nvidia.com/mig-1g.10gb: 1). If your pod requests nvidia.com/gpu: 1, but the node only advertises nvidia.com/mig-1g.10gb, scheduling won’t match. So be consistent in your pod specs. Cluster Autoscaler Currently, MIG-enabled node pools have limited or no autoscaler support on AKS (the cluster autoscaler does not fully account for MIG resources). Scale these node pools manually or via custom logic. If you rely heavily on auto-scaling, consider using a standard GPU node pool (no MIG) or carefully plan capacity to avoid needing dynamic scaling for MIG pools. Monitoring The GPU Operator deploys DCGM exporter by default, which can collect MIG-specific metrics. Integrate with Prometheus + Grafana for GPU usage dashboards. MIG slices are typically identified by unique device IDs in DCGM. You can see which MIG slices are busier than others, memory usage, etc. Node Image Upgrades Because you’re skipping the driver install from AKS, ensure you keep your GPU driver DaemonSet or Operator up to date. If you do a node image upgrade (AKS version upgrade), the OS might change, requiring a recompile or a matching driver version. The GPU Operator normally handles this seamlessly by re-installing the driver on the new node image. Test your upgrades in a staging cluster if possible, especially with new AKS releases or driver versions. Handling Multiple Node Pools Many users create one node pool with full GPUs (no MIG) for large jobs, and another MIG-enabled node pool for smaller parallel workloads. You can do so easily by repeating the steps above for each node pool, specifying different MIG profiles. References MIG User Guide NVIDIA GPU Operator with Azure Kubernetes Service ND-H100-v5 sizes series Create a multi-instance GPU node pool in Azure Kubernetes Service (AKS)
Agustinmantaras
Mar 28, 2025 Place AI - Azure AI services Blog
971Views
2likes
0Comments
Azure at KubeCon Europe 2025 | London, UK - April 1-4
Are you ready for KubeCon + CloudNativeCon Europe 2025? We are thrilled to join the community in London, UK, from April 1-4, 2025, for an exciting lineup of events and activities. As a Diamond Sponsor, Microsoft Azure is set to showcase the latest innovations in Kubernetes, AI, and open-source. Read on for the many ways to connect with our team during the event: Azure Day with Kubernetes (April 1st): (This is not a prank!) We are kicking things off with our Azure Day with Kubernetes pre-day event, focused on all things cloud-native and intelligent apps with Kubernetes on Azure. Morning - Presentations: Start the day by learning about the latest on AKS. Learn how AKS streamlines the deployment and management intelligent applications, and gain insights into the best practices across key areas. We will also be joined by a special guest speaker from Adobe! Afternoon – Choose Your Own Adventure Attendees have a choice of two activities in the afternoon: AKS Labs: Get hands-on with an instructor-led AKS lab and put some of the concepts and ideas from the morning session to practice. Deep-dive roundtables: Join AKS experts for deep-drive roundtables on a variety of topics, including security, autoscaling, performance, AI, & more. Participants will have the opportunity to ask questions and discuss their most pressing concerns with our subject-matter experts. Evening – Pints on us! Space is limited and available on a first-come, first-served basis, so sign up for Azure Day with Kubernetes - London 2025 as soon as possible. KubeCon + CloudNativeCon (April 2-4): Don’t miss the Microsoft keynote on Evolving the Kubernetes user experience, by Andrew Randall on Wednesday, April 2nd. Ryan Zhang will be at the KubeCon Demo Theatre showcasing KubeFleet - The Future of Multi-Cluster Kubernetes Application Management on Thursday, April 3rd. Check out sessions across the event by Microsoft engineers on diverse topics – the full session list is included below! Catch our teams in the Project Pavilion supporting some of the many open-source project kiosks, including Inspektor Gadget, Headlamp, Notary Project, Flatcar, ORAS, Ratify, and Istio. Brendan Burns, Kubernetes co-founder and Microsoft CVP, will share his thoughts on the latest developments and key Microsoft announcements related to open-source, cloud native, and intelligent application development in his KubeCon Europe blog on April 2nd. And of course, swing by our Microsoft Azure booth #N150 from April 2-4! We'll have short sessions and demos on all things cloud native and AI, an AI activation zone with Microsoft Designer, and some sweet swag up for grabs. Hot tip: join us for our live Kubernetes Trivia Show at the Microsoft Azure booth during the KubeCrawl on Wednesday to win exclusive swag! Read on below for full details on our KubeCon sessions and booth theater presentations: Sponsored Keynote Date: Wed April 2, 2025  Start Time: 9:49 AM BST Room: ICC Auditorium Title: Evolving the Kubernetes User Experience Speaker: Andrew Randall Abstract: Kubernetes is known for being challenging; getting started with such a complex distributed system can be intimidating. For both new and experienced users, it's hard getting started from scratch, handling multi-cluster deployments, and integrating various tools and projects required in a modern Kubernetes environment. So how do we address this? With an intuitive, out-of-the-box experience that simplifies the setup and management of local Kubernetes environments. CNCF project Headlamp's adaptability allows it to offer a great day-one experience for end users and serve as a foundation for vendors to build upon and integrate with other CNCF projects. Bringing Headlamp's capabilities to a desktop experience provides a modern, feature-rich user interface that works to streamline the user experience and make Kubernetes more accessible to all. KubeCon Theatre Demo Date: Thu April 3, 2025  Start Time: 1:15 PM BST Room: Level 1 | Solutions Showcase | Hall Entrances S8 - S9 | Demo Theater Title: KubeFleet: The Future of Multi-Cluster Kubernetes Application Management Speaker: Ryan Zhang Abstract: As organizations increasingly deploy applications across multiple Kubernetes clusters, they face operational challenges. KubeFleet, an open-source project, offers a comprehensive framework to streamline multi-cluster application management. Feature Highlights: Rich Scheduling Capability: Supports k8s native scheduling operations and introduces multi-cluster policies like pick N clusters or pick all clusters. Metrics-Based Scheduling: Sophisticated placement strategies based on internal (GPU, CPU, memory) and external metrics (cost, IP availability). Built-in Continuous Deployment Strategies: Includes rolling update strategies similar to k8s native deployment and stage-based continuous deployment with wait and approval between stages. Join us in this demo session to see how KubeFleet provides advanced scheduling and built-in continuous deployment strategies to manage your multi-cluster applications. KubeCon Europe 2025 Sessions with Microsoft Speakers Speaker Title Andrew Randall Microsoft keynote: Evolving the Kubernetes user experience Ryan Zhang Microsoft demo: KubeFleet - The Future of Multi-Cluster Kubernetes Application Management Toddy Mladenov Panel: Native OCI Integration - Leveraging Argo CD Source Content in a Brand New Way Keith Mattix, Mitch Connors A New Frontier: Sidecarless Service Mesh on Windows With Istio's Ambient Mesh Keith Mattix Towards a Standardized Identity-based Authorization in Kubernetes Ashna Mehrotra Project Lightning Talk: Project Copacetic Feynman Zhou Project Lightning Talk: ORAS: Create and Distribute a Multi-platform Image with Security Posture Yi Zha Project Lightning Talk: Protect your Kubernetes Clusters with Ratify and Attestations Yi Zha Project Lightning Talk: Notary: Securing Binary Artifacts with Fine-grained Control Esteban Rey Streamlined Efficiency: Unshackling Kubernetes Image Volumes for Rapid AI Model and Dataset Loading Ganeshkumar Ashokavardhana Transparent, Infra-Level Checkpoint and Restore for Resilient AI/ML Workloads at Scale Ryan Zhang SIG-Multicluster Intro and Deep Dive Ryan Zhang How the SIG-Multicluster API Specifications Are Used for Real World Multicluster Management Sachi Desai Beyond Kubernetes: Adapting To Specialized Application Workloads Rita Zhang, Standa Láznička The Key To Security: Externalized Service Account Key Management David Justice Building WebAssembly Like It's 2011 David Justice Wasm I Right or Wasm I Wrong? a Review of the Wasm Ecosystem Jeremy Rickard Signed, Sealed, Delivered - Sign and Verify All the Things Jiaxiao "Joe" Zhou Containerd: Project Update and Deep Dive Rita Zhang A Practical Guide To Kubernetes Policy as Code Bridget Kromhout SIG Cloud Provider Deep Dive: Testing Cloud Controller Managers Danielle Tal Discover CNCF TAG Runtime: Advancing Cloud-Native Innovation from AI to Edge Mike Morris Taming the Traffic: Selecting the Perfect Gateway Implementation for You Yi Zha Notary Project: The Key To Secure Software Supply Chain Joaquim Rocha, Rene Dudfield Contribfest: Make Your Own UI for Kubernetes With Headlamp Mathieu Tortuyaux Contribfest: Extending Image Based Systems Using Systemd System Extensions Michael Friese, Qasim Sarfraz, Burak Ok Contribfest: Kubernetes Observability Simplified: Build, Debug & Monitor With Inspektor Gadget Jeremy Rickard An Imperfect Guide To Impact and Intent Neha Aggarwal That’s Just My Cup of Tea: Configuring Cilium for Performance and Scale Will Daly, Tamilmani Manoharan Improving Network Efficiency: The Power of Integrating Delegated IPAM With Cilium Trask Stalnaker OpenTelemetry Project Update Jonathan Smith How Millennium Bcp Leverages Radius To Empower Developer + Operator Collaboration Ciprian Hacman Etcd V3.6.0 and Etcd-operator V0.1.0 Aurélien Bombo Trust No One: Secure Storage With Confidential Containers Microsoft Booth Theatre Sessions Wed Apr 2 How to keep your applications running even when your clusters are not working To Mesh or not To Mesh Best Practices to Manage your AKS Platform with AI-powered observability Headlamp as a UI for all Kubernetes users and projects Getting started with GitHub CoPilot Lose sleep over cluster upgrades? Fret not, Azure Kubernetes Fleet Manager has got your back eBPF-Powered Observability Beyond Azure: A Multi-Cloud Perspective with Retina Hyperlight: Hardware-Protect your Serverless Anywhere At Scale Managing Kubernetes with Azure Linux Cloud native security made easy Thu Apr 3 GenAI workload scheduling & Orchestration in K8S utilizing latest GPUs and 3P scheduler integration on K8S Unlocking the Full Potential of AKS with Traefik: Architecture, Implementation, and Innovation DNS Issues? Here’s how you can use open source tools to debug failed DNS requests in your AKS Cluster Application deployment using AKS Automatic Ensuring Workload Reliability with Every AKS Deployment Karpenter for Resiliency Optimizing cost and efficiency of AI and GPU workloads on AKS Fri Apr 4 Upgrade & Maintenance on Karpenter Azure Container Storage: Latest Next-Generation Persistent Storage for Kubernetes Image assurance from Build to Runtime There will also be a wide variety of demos running at our booth throughout the show – be sure to swing by to chat with the team. We look forward to seeing you at KubeCon Europe 2025 in London! Psst! Local or coming in to London early? You can also catch the Microsoft team at: Cloud Native Rejekts on March 30-31st Maintainer Summit on March 31st And as a sponsor of Istio Day on April 1st
coryskimming
Mar 27, 2025 Place Apps on Azure Blog
757Views
1like
0Comments
Monitor OpenAI Agents SDK with Application Insights
As AI agents become more prevalent in applications, monitoring their behavior and performance becomes crucial. In this blog post, we'll explore how to monitor the OpenAI Agents SDK using Azure Application Insights through OpenTelemetry integration. Enhancing OpenAI Agents with OpenTelemetry The OpenAI Agents SDK provides powerful capabilities for building agent-based applications. By default, the SDK doesn't emit OpenTelemetry data, as noted in GitHub issue #18. This presents an opportunity to extend the SDK's functionality with robust observability features. Adding OpenTelemetry integration enables you to: Track agent interactions across distributed systems Monitor performance metrics in production Gain insights into agent behaviour Seamlessly integrate with existing observability platforms Fortunately, the Pydantic Logfire SDK has implemented an OpenTelemetry instrumentation wrapper for OpenAI Agents. This wrapper allows us to capture telemetry data and propagate it to an OpenTelemetry Collector endpoint. How It Works The integration works by wrapping the OpenAI Agents tracing provider with a Logfire-compatible wrapper that generates OpenTelemetry spans for various agent activities: Agent runs Function calls Chat completions Handoffs between agents Guardrail evaluations Each of these activities is captured as a span with relevant attributes that provide context about the operation. Implementation Example Here's how to set up the Logfire instrumentation in your application: import logfire from openai import AsyncAzureOpenAI from agents import set_default_openai_client, set_tracing_disabled # Configure your OpenAI client azure_openai_client = AsyncAzureOpenAI( api_key=os.getenv("AZURE_OPENAI_API_KEY"), api_version=os.getenv("AZURE_OPENAI_API_VERSION"), azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT"), azure_deployment=os.getenv("AZURE_OPENAI_DEPLOYMENT") ) # Set as default client and enable tracing set_default_openai_client(azure_openai_client) set_tracing_disabled(False) # Configure OpenTelemetry endpoint os.environ["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"] = "http://0.0.0.0:4318/v1/traces" # Configure Logfire logfire.configure( service_name='my-agent-service', send_to_logfire=False, distributed_tracing=True ) # Instrument OpenAI Agents logfire.instrument_openai_agents() Note: The send_to_logfire=False parameter ensures that data is only sent to your OpenTelemetry collector, not to Logfire's cloud service. Environment Variables: The OTEL_EXPORTER_OTLP_TRACES_ENDPOINT environment variable tells the Logfire SDK where to send the OpenTelemetry traces. If you're using Azure Container Apps with the built-in OpenTelemetry collector, this variable will be automatically set for you. Similarly, when using AKS with auto-instrumentation enabled via the OpenTelemetry Operator, this environment variable is automatically injected into your pods. For other environments, you'll need to set it manually as shown in the example above. Setting Up the OpenTelemetry Collector To collect and forward the telemetry data to Application Insights, we need to set up an OpenTelemetry Collector. There are two ways to do this: Option 1: Run the Collector Locally Find the right OpenTelemetry Contrib Releases for your processor architecture at: https://github.com/open-telemetry/opentelemetry-collector-releases/releases/tag/v0.121.0 Only Contrib releases will support Azure Monitor exporter. ./otelcol-contrib --config=otel-collector-config.yaml Option 2: Run the Collector in Docker docker run --rm \ -v $(pwd)/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml \ -p 4318:4318 \ -p 55679:55679 \ otel/opentelemetry-collector-contrib:latest Collector Configuration Here's a basic configuration for the OpenTelemetry Collector that forwards data to Azure Application Insights: receivers: otlp: protocols: http: endpoint: "0.0.0.0:4318" exporters: azuremonitor: connection_string: "InstrumentationKey=your-instrumentation-key;IngestionEndpoint=https://your-region.in.applicationinsights.azure.com/" maxbatchsize: 100 maxbatchinterval: 10s debug: verbosity: basic service: pipelines: traces: receivers: [otlp] exporters: [azuremonitor, debug] Important: Replace connection_string with your actual Application Insights connection string. What You Can Monitor With this setup, you can monitor various aspects of your OpenAI Agents in Application Insights: Agent Performance: Track how long each agent takes to process requests Model Usage: Monitor which AI models are being used and their response times Function Calls: See which tools/functions are being called by agents Handoffs: Track when agents hand off tasks to other specialized agents Errors: Identify and diagnose failures in agent processing End-to-End Traces: Follow user requests through your entire system Example Trace Visualisation In Application Insights, you can visualise the traces as a hierarchical timeline, showing the flow of operations: Known Issue: Span Name Display in Application Insights When using LogFire SDK 3.8.1 with Application Insights, you might notice that span names appear as message templates (with regular expressions) instead of showing the actual agent or model names. This makes it harder to identify specific spans in the Application Insights UI. Issue: In the current implementation of LogFire SDK's OpenAI Agents integration (source code), the message template is used as the span's name, resulting in spans being displayed with placeholders like {name!r} or {gen_ai.request.model!r} instead of actual values. Temporary Fix Until LogFire SDK introduces a fix, you can modify the /logfire/_internal/integrations/openai_agents.py file to properly format the span names. This is after pip install logfire, the file will usually be at venv/lib/python3.11/site-packages/logfire/_internal/integrations/openai_agents.py Replace the span creation code around line 100: Original code logfire_span = self.logfire_instance.span( msg_template, **attributes_from_span_data(span_data, msg_template), **extra_attributes, _tags=['LLM'] * isinstance(span_data, GenerationSpanData), ) Modified code with setting Span name as message attributes = attributes_from_span_data(span_data, msg_template) message = logfire_format(msg_template, dict(attributes or {}), NOOP_SCRUBBER) logfire_span = self.logfire_instance.span( msg_template, _span_name=message, **attributes, **extra_attributes, _tags=['LLM'] * isinstance(span_data, GenerationSpanData), ) This change formats the message template with actual values and sets it as the span name, making it much easier to identify spans in the Application Insights UI. After applying this fix, your spans will display meaningful names like "Chat completion with 'gpt-4o'" instead of "Chat completion with {gen_ai.request.model!r}". Limitation: Even after applying this fix, HandOff spans will still not show the correct to_agent field in the span name. This occurs because the to_agent field is not set during initial span creation but later in the on_ending method of the LogfireSpanWrapper class: @dataclass class LogfireSpanWrapper(LogfireWrapperBase[Span[TSpanData]], Span[TSpanData]): # ... def on_ending(self): # This is where to_agent gets updated, but too late for the span name # ... Until LogFire SDK optimizes this behavior, you can still see the correct HandOff values by clicking on the span and looking at the logfire.msg property. For example, you'll see "Handoff: Customer Service Agent -> Investment Specialist" in the message property even if the span name doesn't show it correctly. Auto-Instrumentation for AKS Azure Kubernetes Service (AKS) offers a codeless way to enable OpenTelemetry instrumentation for your applications. This approach simplifies the setup process and ensures that your OpenAI Agents can send telemetry data without requiring manual instrumentation. How to Enable Auto-Instrumentation To enable auto-instrumentation for Python applications in AKS, you can add an annotation to your pod specification: annotations: instrumentation.opentelemetry.io/inject-python: 'true' This annotation tells the OpenTelemetry Operator to inject the necessary instrumentation into your Python application. For more details, refer to the following resources: Microsoft Learn: Codeless application monitoring for Kubernetes OpenTelemetry Docs: Automatic Instrumentation for Kubernetes Built-in Managed OpenTelemetry Collector in Azure Container Apps Azure Container Apps provides a built-in Managed OpenTelemetry Collector that simplifies the process of collecting and forwarding telemetry data to Application Insights. This eliminates the need to deploy and manage your own collector instance. Setting Up the Managed Collector When you enable the built-in collector, Azure Container Apps automatically sets the OTEL_EXPORTER_OTLP_ENDPOINT environment variable for your applications. This allows the Logfire SDK to send traces to the collector without any additional configuration. Here's an example of enabling the collector in an ARM template: { "type": "Microsoft.App/containerApps", "properties": { "configuration": { "dapr": {}, "ingress": {}, "observability": { "applicationInsightsConnection": { "connectionString": "InstrumentationKey=your-instrumentation-key" } } } } } For more information, check out these resources: Microsoft Learn: OpenTelemetry agents in Azure Container Apps Tech Community: How to monitor applications by using OpenTelemetry on Azure Container Apps Conclusion Monitoring OpenAI Agents with Application Insights provides valuable insights into your AI systems' performance and behavior. By leveraging the Pydantic Logfire SDK's OpenTelemetry instrumentation and the OpenTelemetry Collector, you can gain visibility into your agents' operations and ensure they're functioning as expected. This approach allows you to integrate AI agent monitoring into your existing observability stack, making it easier to maintain and troubleshoot complex AI systems in production environments. Resources Implementation can be found at https://github.com/hieumoscow/azure-openai-agents References: OpenAI Agents Python SDK GitHub Issue: OpenAI Agents Logging OpenTelemetry Collector Documentation Azure Application Insights Documentation Codeless application monitoring for Kubernetes OpenTelemetry Automatic Instrumentation for Kubernetes OpenTelemetry agents in Azure Container Apps How to monitor applications using OpenTelemetry on Azure Container Apps
hieunhu
Mar 18, 2025 Place AI - Azure AI services Blog
1.4KViews
1like
2Comments
Azure Platform Metrics for AKS Control Plane Monitoring
Azure Kubernetes Service (AKS) now offers free platform metrics for monitoring your control plane components. This enhancement provides essential insights into the availability and performance of managed control plane components, such as the API server and etcd. In this blog post, we'll explore these new metrics and demonstrate how to leverage them to ensure the health and performance of your AKS clusters. What's New? Previously, detailed control plane metrics were only available through the paid Azure Managed Prometheus feature. Now, these metrics are automatically collected for free for all AKS clusters and are available for creating metric alerts. This democratizes access to critical monitoring data and helps all AKS users maintain more reliable Kubernetes environments. Available Control Plane Metrics The following platform metrics are now available for your AKS clusters: Name Display Name Description apiserver_memory_usage_percentage API Server (PREVIEW) Memory Usage Percentage Maximum memory percentage (based off current limit) used by API server pod across instances apiserver_cpu_usage_percentage API Server (PREVIEW) CPU Usage Percentage Maximum CPU percentage (based off current limit) used by API server pod across instances etcd_memory_usage_percentage ETCD (PREVIEW) Memory Usage Percentage Maximum memory percentage (based off current limit) used by ETCD pod across instances etcd_cpu_usage_percentage ETCD (PREVIEW) CPU Usage Percentage Maximum ETCD percentage (based off current limit) used by ETCD pod across instances etcd_database_usage_percentage ETCD (PREVIEW) Database Usage Percentage Maximum utilization of the ETCD database across instances Accessing the New Platform Metrics The metrics are automatically collected and available in the Azure Monitor Metrics explorer. Here's how to access them: Navigate to your AKS cluster in the Azure portal Select "Metrics" from the monitoring section In the Metric namespace dropdown, choose the Metric Namespace as Container Service and Metric as any of the metrics mentioned above e.g. API Server Memory Utilization. You can also choose your desired aggregation (between Avg or Max) and timeframe. You'll now see the control plane metrics available for selection: These metrics can also be retrieved through the platform metrics API or exported to other destinations. Understanding Key Control Plane Metrics API Server Memory Usage Percentage The API server is the front-end for the Kubernetes control plane, processing all requests to the cluster. Monitoring its memory usage is critical because: High memory usage can lead to API server instability and potential outages Memory pressure may cause request latency or timeouts Sustained high memory usage indicates potential scaling issues A healthy API server typically maintains memory usage below 80%. Values consistently above this threshold warrant investigation and potential remediation. A common pitfall is to install an operator that doesn't scale well and it thrashes API server with unreasonable amount of calls. There could also be bugs which could leak large objects and cause API server to run out of memory. A good practice is to monitor and alert on API server memory usage as you deploy new daemonsets. AKS doesn't publish its absolute limits because those are frequently altered as hardware choices and our design improves to better scale with workloads. To investigate further into the issue, follow the guide here. etcd Database Usage Percentage etcd serves as the persistent storage for all Kubernetes cluster data. The etcd_database_usage_percentage metric is particularly important because: etcd performance dramatically degrades as database usage approaches capacity High database utilization can lead to increased latency for all cluster operations Database size impacts backup and restore operations Best practices suggest keeping etcd database usage below 2GB (absolute usage) to ensure optimal performance. While AKS provides a larger database size, when usage exceeds this threshold, it usually affects the performance of the Control Plane. You can clean up unnecessary resources, reduce watch operations, and implement resource quota and limits. The Diagnose and Solve experience in Azure Portal has detailed insights on the cause of the etcd database saturation. To investigate this issue further, follow the guide here. Setting Up Alerts for Control Plane Metrics To proactively monitor your control plane, you can set up metric alerts: Navigate to your AKS cluster in the Azure Portal Select "Alerts" from the monitoring section Click on "Create" and select "Alert Rule" Select your subscription, resource group, and resource type "Kubernetes service" in the Scope (selected by default) and click on See all signals in Conditions Configure signal logic: Select one of the control plane metrics (e.g., "API Server Memory Usage Percentage") Set the condition (e.g., "Greater than") Define the threshold (e.g., 80%) Specify the evaluation frequency and window Define actions to take when the alert triggers Name and save your alert rule Example Alert Configurations API Server Memory Alert: Signal: apiserver_memory_usage_percentage Operator: Greater than Threshold: 80% Window: 5 minutes Frequency: 1 minute Severity: 2 (Warning) ETCD Database Usage Alert: Signal: etcd_database_usage_percentage Operator: Greater than Threshold: 75% Window: 15 minutes Frequency: 5 minutes Severity: 2 (Warning) You can also create alerts through CLI, PowerShell or ARM templates Conclusion The introduction of free Azure platform metrics for AKS control plane components represents a enhancement to the monitoring capabilities available to all AKS users. By leveraging these metrics, particularly API server memory usage and etcd database usage percentages, you can ensure the reliability and performance of your Kubernetes environments without additional cost. Start using these metrics today to gain deeper insights into your AKS clusters and set up proactive alerting to prevent potential issues before they impact your applications. Learn More For more detailed information, refer to the following documentation: Monitor the control plane List of platform metrics in AKS Troubleshoot API server and etcd problems in AKS
aritraghoshmicrosoft
Mar 14, 2025 Place Apps on Azure Blog
817Views
0likes
0Comments
Deploy Datadog Agent to AKS clusters with Datadog Azure Native ISV Service
Having clear visibility and monitoring into your Kubernetes environment is crucial for maintaining the health and performance of your applications. Effective monitoring helps detect issues early, optimize resource usage, and ensure your applications run smoothly. In complex environments, the sheer volume of containers being orchestrated can make monitoring cluster health challenging, which can in turn slow response times to critical incidents and create bottlenecks around long-term optimizations. Enter Datadog, which is a cloud-scale monitoring and security platform that aggregates data across the application stack. With over 850 integrations, it enables customers to monitor the performance of a wide range of technologies, providing a unified platform with centralized visibility for faster troubleshooting on dynamic architectures. To offer a more seamless experience for customers, the Datadog Azure Native ISV Service is a powerful integration between Microsoft Azure and Datadog which allows you to monitor your Azure workloads with Datadog to deliver comprehensive insights into the performance of your tech stack. This native offering is available on both Azure Marketplace and Azure portal and enables you to set up, configure, and deploy a Datadog resource on Azure similarly to creating any other native Azure resources like VMs, App Services etc. It allows you to quickly monitor and diagnose issues with your Azure resources by automatically sending logs and metrics to your Datadog organization. This year, Datadog also announced the availability of the . For Kubernetes environments specifically, Datadog’s integration with Azure Kubernetes Service (AKS) provides complete visibility into your AKS clusters. Once you’ve enabled the integration and deployed the Datadog Agent to your clusters, Datadog automatically begins collecting metrics and logs from your entire AKS setup and organizing them into high-level visualizations. Datadog also provides an out-of-the-box dashboard for AKS that enables customers to immediately visualize the health and performance of their AKS clusters. This dashboard organizes and highlights the most critical information from standard AKS metrics, while also incorporating log data to provide observability into the control plane. Datadog Agent on Kubernetes Apps Marketplace The Datadog Agent can be deployed to AKS using Kubernetes Apps, an offering on Azure Marketplace that features our partners, like Datadog, in the cloud native space. As a user, you can choose from a rich catalog of validated partner and open-source Kubernetes solutions which you can deploy to your AKS and Arc-connected clusters with integrated Azure billing and easy lifecycle management capabilities. You can quickly procure Kubernetes solutions through the trusted procurement channels of Azure Marketplace, with eligible purchases contributing 100% toward your Microsoft Azure Consumption Commitment (MACC), helping you optimize your committed spend. To take the partnership between Microsoft and Datadog further, the Datadog agent can now be installed on AKS resources via Datadog Azure Native ISV Service. How to deploy Datadog Agent on AKS via native integration Datadog Azure Native ISV Service provides out of the box Azure monitoring with seamless flow of observability data from your Azure workloads which you can monitor in real time on the Datadog portal. Once the Datadog resource is deployed on Azure, logs and metrics start to flow almost immediately. Another plus of this integration is that it makes it frictionless to deploy Datadog agent on to several Azure resources directly from Azure portal. We are now happy to extend this capability to your Azure Kubernetes Service (AKS) clusters through Azure portal which makes it easy to automate the process of agent deployment. Now you can install and un-install the Datadog extension on multiple AKS clusters via Azure portal in a few clicks. Follow this step-by-step guide below to deploy the Datadog extension on AKS resources via Azure Native Datadog Service. Create a Datadog resource You can create a Datadog resource in Azure by purchasing Datadog or setting up a trial environment. Once the resource is deployed, the Datadog environment is automatically setup. Deploy the Datadog Agent through Azure portal The Azure Kubernetes Service blade is available on the left panel of the Datadog resource. By clicking on it, you can view a list of all AKS resources in your subscription. Select all the resources you want to deploy the Datadog AKS extension to manually or use the filters that suit your needs. You can also select all the resources by clicking on select all checkbox. There is no need to go to each cluster and install manually. Once selected, click on Install Extension and wait for successful deployment. When the agent is successfully installed, the agent status and version will be displayed against the AKS resource. You can uninstall the extension by clicking on the Uninstall Extension button. Monitor your Azure Kubernetes Service clusters with Datadog There you have it – an easy method to deploy Datadog Agent on your AKS cluster. Now you can use the single sign on link present on the overview blade to access your Datadog environment or just continue to install Agent on other Azure services. Once on the Datadog portal, you can view the detailed insights of your Kubernetes clusters with the help of dedicated dashboards. Why use Datadog Azure Native ISV? Integrated onboarding: Customers can create a Datadog resource and manage the integration through the Azure portal. Unified billing: Datadog costs are reported through Azure monthly bill. Single sign-on to Datadog: Customers don't need a separate authentication for the Datadog portal. Log forwarder: Automated forwarding of subscription activity and resource logs to Datadog. Metrics collection: Automatically send all Azure resource metrics to Datadog. Datadog agent deployment: Provides unified management experience of Datadog agents. Install and uninstall Datadog agents as extensions on Virtual Machines and Azure App Practical Use cases for Datadog + AKS Datadog helps Azure customers make sense of telemetry data across their technology stack. The top use cases that Azure customers rely on Datadog to solve are: migration, modernization, and tool/cost optimization. Migration: Datadog provides deep visibility into complex Azure environments. Further, Datadog is cloud-agnostic and can ingest data from on-prem and hybrid cloud environments. Datadog tracks and secures every stage of your cloud migration, helping customers migrate with confidence and ease Modernization: Datadog helps Azure customers modernize their stack via Azure capabilities like Azure Kubernetes Service (AKS). Datadog provides full visibility into modern and legacy applications in one place. Azure customers report an improvement in performance, agility, and resilience with Datadog. Tool/Cost Optimization: Many organizations turn to Datadog because their teams are experiencing tool sprawl and total cost of ownership challenges from multiple siloed monitoring tools. Datadog addresses both challenges by enabling customers to view all of their data in a single, unified platform. Azure customers can also use Datadog to analyze and optimize their cloud spend. Additional Links Azure Blog: Explore the latest features for Datadog–An Azure Native ISV Service: https://azure.microsoft.com/en-us/blog/explore-the-latest-features-for-datadog-an-azure-native-isv-s... Azure Marketplace: Datadog: https://azuremarketplace.microsoft.com/en-us/marketplace/apps/datadog1591740804488.dd_liftr_v2?tab=o... Benefits of Kubernetes Apps on the Marketplace: Deploy Smarter, Scale Faster – Secure, AI-Ready, Cost-Effective Kubernetes Apps at Your Fingertips! ... Get Started with Datadog - An Azure Native ISV Service: https://learn.microsoft.com/en-us/azure/partner-solutions/datadog/overview Datadog Blog: Datadog-Microsoft Partnership: Azure Native ISV Service https://www.datadoghq.com/blog/azure-datadog-partnership/ Datadog’s AKS Integration: https://www.datadoghq.com/blog/azure-kubernetes-service-cluster-visibility-with-aks-dashboard/ Datadog agent ok AKS Marketplace: https://www.datadoghq.com/blog/datadog-aks-cluster-extension/ Leave a feature request on our feedback forum We value customer feedback and plan feature suggestions into our roadmap. Let us know how we can enhance the value you get from Datadog Azure Native ISV by requesting a feature on the Developer Community. Alternatively, you can utilize “suggest a feature” option available on the Azure portal.
bobmital
Mar 06, 2025 Place Apps on Azure Blog
383Views
1like
0Comments
Retrieving Azure App Service Deployment Center Events - Monitoring
Hello Team, I would like to know how to retrieve Azure App Service Deployment Center events. Specifically, I’m looking to integrate a webhook to capture trigger and deployment events from the Deployment Center. Thanks, Vinoth_Azure
Vinoth_Azure
Feb 24, 2025 Place Apps on Azure
77Views
0likes
2Comments