Troubleshoot health probe failures with Azure Load Balancer Health Status
In today's fast-paced cloud computing environment, maintaining the optimal performance and reliability of your applications is crucial. Azure Load Balancer's Health Status feature , now generally available to customers, significantly simplifies this task by providing detailed health information about your backend instances without the need to file a support ticket. This tool offers invaluable insights into the health state of each backend instance and the specific reasons behind their status, whether user-triggered or platform-triggered. By leveraging this feature, customers can proactively address issues, ensure minimal downtime, and enhance the overall user experience, all while reducing reliance on support services. What is Health Status? Health Status is an Azure Load Balancer feature that gives you detailed health information about the backend instances connected to your Azure Load Balancer’s backend pool. Each status is linked to your load balancing rules and provides two key insights: the health state of each backend instance and the reasoning behind its state. The health state indicates whether your backend instance is healthy ("Up") or unhealthy ("Down"). The reasoning behind these states is explained through reason codes, which fall into two categories: User Triggered Reason Codes and Platform Triggered Reason Codes. User Triggered Reason Codes are based on how you configured your load balancer setup and can be addressed by you. Platform Triggered Reason Codes are based on the Azure Load Balancer platform and cannot be addressed by you. For more information about the different reason codes, view our public documentation. Why use Health Status? In the past, customers were not provided with insights into why their backend instances were deemed healthy or unhealthy. To access this crucial information, customers often had to follow troubleshooting procedures such as taking packet captures or going through the process of creating a support ticket, relying on support engineers to identify the cause of a failed health probe. This process was not only complex and time-consuming but also incurred additional costs and added significant management overhead. Now, with the Health Status feature, customers can easily access real-time health information of their backend instances. This empowers them to make swift and informed decisions, minimizing downtime, reducing support costs, and enhancing the overall user experience. By leveraging these insights, customers can proactively manage their environment and ensure optimal performance. Retrieving Health Status Health Status can be easily retrieved on a per load balancing rule basis. To retrieve Health Status: Sign in to the Azure Portal and search for "Load balancers". Select your load balancer and navigate to "Load balancing rules" under Settings. View the health status of the rule by clicking “View details” value of the corresponding rule. Refresh button can be used to get the latest status. Figure 1: Sample Health Status in Azure Portal Contoso's Utilization of Health Status for Game Server Maintenance Let’s explore how one of our customers, Contoso, uses the Health Status feature for efficient decision-making and troubleshooting. Who is Contoso and what is their issue Contoso, a prominent name in the gaming industry, has been leveraging Azure Load Balancer to distribute traffic to their highly popular game server hosted on Azure Virtual Machine Scale Sets. Their users love using Contoso’s servers due to the reliability and performance achieved on them. Recently, Contoso encountered an issue where one of their game servers became unhealthy, leading to disruptions in the gaming experience for their users. How Health Status resolved their issue Thanks to the Azure Load Balancer Health Status feature, the Contoso team was able to quickly navigate to the Load balancing rule page in Portal to view the health status of the unhealthy virtual machine instance. By doing so, they retrieved detailed insights into why their game server was marked unhealthy. This real-time information highlighted “the backend instance was unhealthy due to Admin State set to Down”. Armed with this crucial data, Contoso's Network team swiftly addressed the configuration issue by toggling the Admin State value of unhealthy server to “None”, thereby restoring the server to a healthy state. After a root cause analysis, it was determined that the previous engineer mistakenly toggled the wrong server to a Down Admin State value when trying to do fixes on another server. Benefits of using Health Status Instead of creating a support ticket and waiting for assistance, they utilized the Health Status feature to diagnose and resolve the problem independently. This proactive approach not only minimized downtime but also reduced support costs and enhanced the overall user experience. Conclusion By incorporating the Health Status feature into their operational workflow, Contoso has been able to make efficient, data-driven decisions and troubleshooting issues promptly, ensuring their gaming services remain robust and reliable for their users. Get Started We are excited to bring the Azure Load Balancer’s Health Status feature to you. This feature provides valuable insights into the health of your backend instances, helping you ensure better troubleshooting for optimal performance and reliability of your applications. For more information and to get started, visit the following links: Overview of health status concepts How to retrieve health status We hope you can take advantage of this feature, and we welcome your feedback. Please feel free to leave a comment below.
Introducing Azure Load Balancer health event logs
We’re thrilled to announce that Azure Load Balancer now supports health event logs! These new logs are published to the Azure Monitor resource log category LoadBalancerHealthEvent and are intended to help you monitor and troubleshoot your load balancer resources. As part of this public preview, you can now receive the following 5 health event types when the associated conditions are met. These health event types are targeted to address the top issues that could affect your load balancer’s health and availability: LoadBalancerHealthEventType Scenario DataPathAvailabilityWarning Detect when the Data Path Availability metric of the frontend IP is less than 90% due to platform issues DataPathAvailabilityCritical Detect when the Data Path Availability metric of the frontend IP is less than 25% due to platform issues NoHealthyBackends Detect when all backend instances in a pool are not responding to the configured health probes HighSnatPortUsage Detect when a backend instance utilizes more than 75% of its allocated ports from a single frontend IP SnatPortExhaustion Detect when a backend instance has exhausted all allocated ports and will fail further outbound connections until ports have been released or more ports are allocated What can I do with Azure Load Balancer health event logs? Create a diagnostic setting to archive or analyze these logs Use Log Analytics querying capabilities Configure an alert to trigger an action based on the generated logs Pictured above is a sample load balancer health event log in Azure portal Why should I use health event logs? Not only do health events give you more insight into the health of your load balancer, you also no longer have to worry about picking a threshold for your metric-based alerts or trying to store difficult to parse metric-based data to identify historical impact to your load balancer resources. As an example, let’s take a look at how customers used to monitor your outbound connectivity health prior to health event logs. Previously in Azure… Context Contoso is leveraging a Standard Public Load Balancer with outbound rules so that their application can connect to public APIs when needed. They are following the recommended guidance and have configured the outbound rules to a dedicated public IP for outbound connections only and have ensure that the backend instances are fully utilizing the 64k available ports by selecting manual port allocation. For their load balancers, they anticipate having at-most, 8 backend instances in a pool at any given time, so they allocate 8k ports to each backend instance using an outbound rule. Problem However, Contoso is still concerned about the risk of SNAT port exhaustion. They also aren’t sure how much traffic they anticipate receiving, or what their traffic patterns will look like. As a result, they want to create an alert to warn them in advance if it looks like any backend instances are close to consuming all of the allocated SNAT ports. Alerting with metrics Using the Used SNAT ports metric, they create an alert that triggers when the metric value exceeds 6k ports, indicated that 75% of the 8k allocated ports have been used. This works, until they receive this alert and decide to add another public IP, doubling the number of allocated ports per backend instance. Now, Contoso needs to update their alert to trigger when the metric value exceeds 12k ports instead. Now: with the HighSnatPortUsage and SnatPortExhaustion events… The team at Contoso learns about Load Balancer’s new health event logs and decide to configure two alerts: Send an email and create an incident whenever the HighSnatPortUsage event is generated, to warn their network engineers that more SNAT ports may need to be allocated to their load balancer’s backend instances Notifies the on-call engineer whenever the SnatPortExhaustion event is generated, to immediately address any potentially critical impact to their applications Now, even when more ports are allocated, Contoso doesn’t have to worry about readjusting their alert rules. What’s next? As part of this public preview announcement, we’re ushering in a new era of health and monitoring improvements for Azure Load Balancer. These five health event types are just the start of empowering you to identify, troubleshoot, and resolve issues related to your resources as quickly as possible. Stay tuned as we add additional health event types to cover other types of scenarios, ranging from providing configuration guidance and best practices, to surfacing warnings when you’re approaching service-related limits. Feel free to leave any feedback you have by leaving comments on this Azure Feedback post, we look forward to hearing from you and are excited for you to try out health event logs. Get started Load balancer health event logs are now rolling all Azure public regions. For more information on the current regional availability, along with more about these logs and how to start collecting and troubleshooting them, take a look at our public documentation.
Build an Azure Logic App to send an alert when the provisioning state changes for your Azure VWAN
Have you ever thought about monitoring your VWAN provisioning state? In today’s dynamic cloud environment, it’s crucial to have a good monitoring system in place for your Azure resources especially your networking resources. In this blog, I am going to show you step-by-step how to create an Azure Logic App that will notify you whenever the provisioning status of your Azure Virtual WAN changes to another state other than “Succeeded”. Note: As of the time I am writing this blog, Microsoft doesn’t support alerts based on Metrics for Azure Virtual WAN. You can still retrieve information for the components that shape up the Virtual WAN for example, VPN site-site connections, BGP peers, virtual hubs, etc. However, you won’t be able to configure diagnostic logs for virtual WAN at this time. Good news — There’s an alternate way to be alerted via an Azure Logic AppIntroducing Azure Gateway Load Balancer: Deploy and scale network virtual appliances with ease
Today, we are pleased to announce the preview of Gateway Load Balancer, a fully managed service enabling you to deploy, scale, and enhance the availability of third party NVAs in Azure. You can add your favorite third party appliance whether it is a firewall, inline DDoS appliance, deep packet inspection system, or even your own custom appliance into the network path transparently – all with a single click.
