Creating a Reliable Notification System for Azure Spot VM Evictions (preempt) events
Introduction Azure Spot VMs offer significant cost savings but come with a trade-off: they can be evicted with minimal notice when Azure needs the capacity back or price change. Building a reliable notification system for these evictions is critical for applications that need to respond gracefully to these events. What are Azure Spot VMs? Azure Spot VMs are virtual machines that use spare capacity in Azure data centers, available at significantly discounted prices compared to regular pay-as-you-go VMs. Microsoft offers this unused capacity at discounts of up to 90% off the standard prices, making Spot VMs an extremely cost-effective option for many workloads. However, there's an important caveat: when Azure needs this capacity back for regular pay-as-you-go customers, your Spot VMs can be evicted (reclaimed) with minimal notice - typically just 30 seconds. This eviction mechanism is what allows Microsoft to offer such deep discounts, as we maintain the flexibility to reclaim these resources when needed. https://azure.microsoft.com/en-gb/products/virtual-machines/spot Benefits of Spot VMs Significant cost savings: The most obvious benefit is the substantial discount, which can be up to 90% off standard VM prices. Same VM types and features: Spot VMs provide the same performance, features, and capabilities as regular VMs - the only difference is the eviction possibility. Ideal for interruptible workloads: For workloads that can handle interruptions, such as batch processing jobs, dev/test environments, or stateless applications, Spot VMs offer enormous value. Flexible sizing options: Spot VMs are available in most VM series and regions, giving you access to a wide range of computing options. Scaling opportunities: The cost savings enable you to run larger clusters or more powerful VMs than might be financially feasible with regular VMs. Effective for burst capacity: When you need additional capacity for temporary workloads, Spot VMs can provide it at minimal cost. Great for fault-tolerant applications: Modern cloud-native applications designed with redundancy and resilience can leverage Spot VMs excellently since they're built to handle node failures. Why Not Just Use Azure Resource Events? A common question is: "Why not simply listen for Azure Resource events like ResourceActionSuccess for VM evictions?" While Azure does emit platform events when resources change state through resource group as source for Azure Event Grid topic subscription, there are several critical limitations when relying on these for Spot VM evictions: Timing issues: By the time a ResourceActionSuccess event is generated for a VM eviction, it is possible that the VM is already being evicted. This gives you no time to perform graceful shutdown procedures. Reliability concerns: These events pass through multiple Azure systems before reaching your event handlers, adding potential points of failure and latency. Ambiguous events: Resource action events don't clearly distinguish between a normal VM shutdown and a Spot VM eviction, making it difficult to trigger the right response. For example: I initially attempted to capture Azure Spot VM eviction events by setting up event notifications on an Azure resource group and publishing them to Service Bus. While this configuration successfully captured some Azure Resource events, it ultimately proved unreliable for eviction monitoring. The solution missed several critical eviction events and, more problematically, could not reliably distinguish between intentional VM shutdowns and actual eviction events. This lack of differentiation made automated response handling impossible, as the system couldn't determine whether a VM was being evicted by Azure or simply stopped through normal administrative actions. Azure resource group as an Event Grid source - Azure Event Grid | Microsoft Learn For these reasons, the most reliable approach is to detect eviction events directly from within the VM using the Azure Instance Metadata Service (IMDS) Scheduled Events API, which is specifically designed to provide advance notice of impending VM state changes. This blog post will guide you through implementing a solution that: Detects Spot VM eviction events from within the VM Formats these events properly Sends them to an Azure Event Grid custom topic Sets up proper event handling downstream Understanding Spot VM Eviction Notices Spot VMs receive eviction notifications approximately 30 seconds before being reclaimed. These notifications are delivered through the Azure Instance Metadata Service (IMDS) Scheduled Events API - an endpoint available from within the VM at http://169.254.169.254/metadata/scheduledevents. When a Spot VM is about to be evicted, a "Preempt" event appears in the Scheduled Events data. Your application needs to poll this endpoint regularly to detect these events in time to take action. https://learn.microsoft.com/en-us/azure/virtual-machines/windows/scheduled-events Solution overview Our solution consists of below components: A custom Event Grid topic to receive and distribute the events - optional if you wish to handle on own from VM A monitoring script running inside the Spot VM - actual script to poll events running on VM Logic to format and send events from the VM to Event Grid Event subscribers that take action when evictions occur A) Setting Up the Event Grid Custom Topic First, create an Event Grid custom topic that will serve as the distribution mechanism for your eviction events - this can be optional if you plan to take actions from VM only like gracefully shutting down any existing processes. You can use below documentation to create custom event grid topic: Custom topics in Azure Event Grid - Azure Event Grid | Microsoft Learn B) Creating a Windows-Based Eviction Monitor For Windows Spot VMs, we'll use below PowerShell to poll preempt events & send it to custom event grid. Create a script file named SpotMonitor.ps1: Powershell script : SpotMonitor.ps1 # Configuration variables - replace with your values $EventGridTopicEndpoint = "https://<EG topic name>.westeurope-1.eventgrid.azure.net/api/events" $EventGridKey = "<EG key>" $CheckInterval = 3 # seconds between checks - feel free to modify as per your requirement $LogFile = "C:\Logs\spot-monitor.log" # Create log directory if it doesn't exist if (-not (Test-Path (Split-Path $LogFile))) { New-Item -ItemType Directory -Path (Split-Path $LogFile) -Force } function Write-Log { param ([string]$Message) $timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss" "$timestamp - $Message" | Out-File -FilePath $LogFile -Append } Write-Log "Starting Spot VM eviction monitor..." while ($true) { try { # Get the VM's metadata including scheduled events $headers = @{"Metadata" = "true"} $scheduledEvents = Invoke-RestMethod -Uri "http://169.254.169.254/metadata/scheduledevents?api-version=2020-07-01" -Headers $headers # Check if there are any events if ($scheduledEvents.Events -and $scheduledEvents.Events.Count -gt 0) { Write-Log "Found $($scheduledEvents.Events.Count) scheduled events" # Get VM metadata for context $vmName = Invoke-RestMethod -Uri "http://169.254.169.254/metadata/instance/compute/name?api-version=2020-09-01&format=text" -Headers $headers $resourceGroup = Invoke-RestMethod -Uri "http://169.254.169.254/metadata/instance/compute/resourceGroupName?api-version=2020-09-01&format=text" -Headers $headers $subscription = Invoke-RestMethod -Uri "http://169.254.169.254/metadata/instance/compute/subscriptionId?api-version=2020-09-01&format=text" -Headers $headers # Process each event foreach ($event in $scheduledEvents.Events) { if ($event.EventType -eq "Preempt") { Write-Log "ALERT: Spot VM preemption detected!" # Extract event details $eventId = $event.EventId $notBefore = $event.NotBefore Write-Log "VM $vmName will be preempted not before $notBefore" # Create Event Grid event as an array (critical for EventGrid schema) $eventGridEvent = @( @{ subject = "/subscriptions/$subscription/resourceGroups/$resourceGroup/providers/Microsoft.Compute/virtualMachines/$vmName" eventType = "SpotVM.Preemption" eventTime = (Get-Date).ToUniversalTime().ToString("o") id = [Guid]::NewGuid().ToString() data = @{ vmName = $vmName resourceGroup = $resourceGroup subscription = $subscription preemptionTime = $notBefore eventId = $eventId eventType = $event.EventType } dataVersion = "1.0" } ) # Convert to JSON - ensuring it stays as an array $eventGridPayload = ConvertTo-Json -InputObject $eventGridEvent -Depth 10 # Send to Event Grid $eventGridHeaders = @{ "Content-Type" = "application/json" "aeg-sas-key" = $EventGridKey } try { $response = Invoke-RestMethod -Uri $EventGridTopicEndpoint -Method Post -Body $eventGridPayload -Headers $eventGridHeaders Write-Log "Successfully sent event to Event Grid" # Take actions to prepare for shutdown Write-Log "Taking actions to prepare for shutdown..." # Example: Stop services gracefully # Stop-Service -Name "YourServiceName" -Force } catch { Write-Log "Failed to send to Event Grid: $_" } } } } } catch { Write-Log "Error checking for events: $_" } # Wait before checking again Start-Sleep -Seconds $CheckInterval } The script above checks for eviction events every 3 seconds by default. You can adjust this polling frequency by changing the "Check_Interval" variable in the script to better match your specific system requirements and performance considerations. More frequent polling provides faster detection but increases resource usage, while less frequent polling reduces overhead but might slightly delay event detection. B) Running monitor script as a scheduler or service For Windows Spot VMs, we'll use PowerShell to create a monitoring service. Run a script file named SpotMonitor.ps1 created in last step: You can use Windows Task Scheduler to run the script at startup or to run as a service and the logs will looks like this: Logs: 2025-03-19 18:48:27 - Starting Spot VM eviction monitor... 2025-03-19 20:04:33 - Found 1 scheduled events 2025-03-19 20:04:33 - ALERT: Spot VM preemption detected! 2025-03-19 20:04:33 - VM anivmnew will be preempted not before Wed, 19 Mar 2025 20:04:47 GMT 2025-03-19 20:04:33 - Sending payload: [ { "eventTime": "2025-03-19T20:04:33.4655660Z", "data": { "eventId": "DE2EC5FA-AF0A-4D59-85D2-677C66A6BC12", "preemptionTime": "Wed, 19 Mar 2025 20:04:47 GMT", "eventType": "Preempt", "resourceGroup": "RG-TEST", "subscription": "azure-sub-id", "vmName": "anivmnew" }, "id": "5d3e6430-dff5-45da-ae90-992e3e342d37", "subject": "/subscriptions/azure-sub-id/resourceGroups/RG-TEST/providers/Microsoft.Compute/virtualMachines/anivmnew", "eventType": "SpotVM.Preemption", "dataVersion": "1.0" } ] 2025-03-19 20:04:33 - Event Grid response: 2025-03-19 20:04:33 - Taking actions to prepare for shutdown... 2025-03-19 20:04:36 - Found 1 scheduled events 2025-03-19 20:04:36 - ALERT: Spot VM preemption detected! 2025-03-19 20:04:36 - VM anivmnew will be preempted not before Wed, 19 Mar 2025 20:04:47 GMT 2025-03-19 20:04:36 - Sending payload: [ { "eventTime": "2025-03-19T20:04:36.6382480Z", "data": { "eventId": "DE2EC5FA-AF0A-4D59-85D2-677C66A6BC12", "preemptionTime": "Wed, 19 Mar 2025 20:04:47 GMT", "eventType": "Preempt", "resourceGroup": "RG-TEST", "subscription": "azure-sub-id", "vmName": "anivmnew" }, "id": "b6152429-f4cb-43b9-8c53-b6ceb08946e5", "subject": "/subscriptions/azure-sub-id/resourceGroups/RG-TEST/providers/Microsoft.Compute/virtualMachines/anivmnew", "eventType": "SpotVM.Preemption", "dataVersion": "1.0" } ] 2025-03-19 20:04:36 - Event Grid response: 2025-03-19 20:04:36 - Taking actions to prepare for shutdown... 2025-03-19 20:04:39 - Found 1 scheduled events 2025-03-19 20:04:39 - ALERT: Spot VM preemption detected! 2025-03-19 20:04:39 - VM anivmnew will be preempted not before Wed, 19 Mar 2025 20:04:47 GMT 2025-03-19 20:04:39 - Sending payload: [ { "eventTime": "2025-03-19T20:04:39.7567285Z", "data": { "eventId": "DE2EC5FA-AF0A-4D59-85D2-677C66A6BC12", "preemptionTime": "Wed, 19 Mar 2025 20:04:47 GMT", "eventType": "Preempt", "resourceGroup": "RG-TEST", "subscription": "azure-sub-id", "vmName": "anivmnew" }, "id": "e0bde6d0-ae27-4c01-8e69-621e57d70f8d", "subject": "/subscriptions/azure-sub-id/resourceGroups/RG-TEST/providers/Microsoft.Compute/virtualMachines/anivmnew", "eventType": "SpotVM.Preemption", "dataVersion": "1.0" } ] 2025-03-19 20:04:39 - Event Grid response: 2025-03-19 20:04:39 - Taking actions to prepare for shutdown... 2025-03-19 20:04:42 - Found 1 scheduled events 2025-03-19 20:04:42 - ALERT: Spot VM preemption detected! 2025-03-19 20:04:42 - VM anivmnew will be preempted not before Wed, 19 Mar 2025 20:04:47 GMT 2025-03-19 20:04:42 - Sending payload: [ { "eventTime": "2025-03-19T20:04:42.8339675Z", "data": { "eventId": "DE2EC5FA-AF0A-4D59-85D2-677C66A6BC12", "preemptionTime": "Wed, 19 Mar 2025 20:04:47 GMT", "eventType": "Preempt", "resourceGroup": "RG-TEST", "subscription": "azure-sub-id", "vmName": "anivmnew" }, "id": "ab7a3b84-bcd8-4651-829e-c57043c54b92", "subject": "/subscriptions/azure-sub-id/resourceGroups/RG-TEST/providers/Microsoft.Compute/virtualMachines/anivmnew", "eventType": "SpotVM.Preemption", "dataVersion": "1.0" } ] 2025-03-19 20:04:42 - Event Grid response: 2025-03-19 20:04:42 - Taking actions to prepare for shutdown... 2025-03-19 20:04:45 - Found 1 scheduled events 2025-03-19 20:04:45 - ALERT: Spot VM preemption detected! 2025-03-19 20:04:45 - VM anivmnew will be preempted not before Wed, 19 Mar 2025 20:04:47 GMT 2025-03-19 20:04:45 - Sending payload: [ { "eventTime": "2025-03-19T20:04:45.9317109Z", "data": { "eventId": "DE2EC5FA-AF0A-4D59-85D2-677C66A6BC12", "preemptionTime": "Wed, 19 Mar 2025 20:04:47 GMT", "eventType": "Preempt", "resourceGroup": "RG-TEST", "subscription": "azure-sub-id", "vmName": "anivmnew" }, "id": "eacfae6b-4ea5-426d-8bc2-659320a7baf0", "subject": "/subscriptions/azure-sub-id/resourceGroups/RG-TEST/providers/Microsoft.Compute/virtualMachines/anivmnew", "eventType": "SpotVM.Preemption", "dataVersion": "1.0" } ] 2025-03-19 20:04:45 - Event Grid response: 2025-03-19 20:04:45 - Taking actions to prepare for shutdown... 2025-03-19 20:04:48 - Found 1 scheduled events 2025-03-19 20:04:49 - ALERT: Spot VM preemption detected! 2025-03-19 20:04:49 - VM anivmnew will be preempted not before Wed, 19 Mar 2025 20:04:47 GMT 2025-03-19 20:04:49 - Sending payload: [ { "eventTime": "2025-03-19T20:04:49.0666732Z", "data": { "eventId": "DE2EC5FA-AF0A-4D59-85D2-677C66A6BC12", "preemptionTime": "Wed, 19 Mar 2025 20:04:47 GMT", "eventType": "Preempt", "resourceGroup": "RG-TEST", "subscription": "azure-sub-id", "vmName": "anivmnew" }, "id": "b2142ee8-9ecf-441d-846e-c8ed663a949e", "subject": "/subscriptions/azure-sub-id/resourceGroups/RG-TEST/providers/Microsoft.Compute/virtualMachines/anivmnew", "eventType": "SpotVM.Preemption", "dataVersion": "1.0" } ] 2025-03-19 20:04:49 - Event Grid response: 2025-03-19 20:04:49 - Taking actions to prepare for shutdown... 2025-03-19 20:04:52 - Found 1 scheduled events 2025-03-19 20:04:52 - ALERT: Spot VM preemption detected! 2025-03-19 20:04:52 - VM anivmnew will be preempted not before Wed, 19 Mar 2025 20:04:47 GMT 2025-03-19 20:04:52 - Sending payload: [ { "eventTime": "2025-03-19T20:04:52.1310990Z", "data": { "eventId": "DE2EC5FA-AF0A-4D59-85D2-677C66A6BC12", "preemptionTime": "Wed, 19 Mar 2025 20:04:47 GMT", "eventType": "Preempt", "resourceGroup": "RG-TEST", "subscription": "azure-sub-id", "vmName": "anivmnew" }, "id": "d9eba318-9773-4e73-a694-dd1c1bf89c10", "subject": "/subscriptions/azure-sub-id/resourceGroups/RG-TEST/providers/Microsoft.Compute/virtualMachines/anivmnew", "eventType": "SpotVM.Preemption", "dataVersion": "1.0" } ] 2025-03-19 20:04:52 - Event Grid response: 2025-03-19 20:04:52 - Taking actions to prepare for shutdown... 2025-03-19 20:04:55 - Found 1 scheduled events 2025-03-19 20:04:55 - ALERT: Spot VM preemption detected! 2025-03-19 20:04:55 - VM anivmnew will be preempted not before Wed, 19 Mar 2025 20:04:47 GMT 2025-03-19 20:04:55 - Sending payload: [ { "eventTime": "2025-03-19T20:04:55.2171546Z", "data": { "eventId": "DE2EC5FA-AF0A-4D59-85D2-677C66A6BC12", "preemptionTime": "Wed, 19 Mar 2025 20:04:47 GMT", "eventType": "Preempt", "resourceGroup": "RG-TEST", "subscription": "azure-sub-id", "vmName": "anivmnew" }, "id": "c358c433-50f5-496d-8823-c2ffddd03390", "subject": "/subscriptions/azure-sub-id/resourceGroups/RG-TEST/providers/Microsoft.Compute/virtualMachines/anivmnew", "eventType": "SpotVM.Preemption", "dataVersion": "1.0" } ] 2025-03-19 20:04:55 - Event Grid response: 2025-03-19 20:04:55 - Taking actions to prepare for shutdown... 2025-03-19 20:04:58 - Found 1 scheduled events 2025-03-19 20:04:58 - ALERT: Spot VM preemption detected! 2025-03-19 20:04:58 - VM anivmnew will be preempted not before Wed, 19 Mar 2025 20:04:47 GMT 2025-03-19 20:04:58 - Sending payload: [ { "eventTime": "2025-03-19T20:04:58.3040422Z", "data": { "eventId": "DE2EC5FA-AF0A-4D59-85D2-677C66A6BC12", "preemptionTime": "Wed, 19 Mar 2025 20:04:47 GMT", "eventType": "Preempt", "resourceGroup": "RG-TEST", "subscription": "azure-sub-id", "vmName": "anivmnew" }, "id": "3eacba95-e05f-41dc-b9e7-1593fe2a71e2", "subject": "/subscriptions/azure-sub-id/resourceGroups/RG-TEST/providers/Microsoft.Compute/virtualMachines/anivmnew", "eventType": "SpotVM.Preemption", "dataVersion": "1.0" } ] 2025-03-19 20:04:58 - Event Grid response: 2025-03-19 20:04:58 - Taking actions to prepare for shutdown... 2025-03-19 20:05:01 - Found 1 scheduled events 2025-03-19 20:05:01 - ALERT: Spot VM preemption detected! 2025-03-19 20:05:01 - VM anivmnew will be preempted not before 2025-03-19 20:05:01 - Sending payload: [ { "eventTime": "2025-03-19T20:05:01.3842973Z", "data": { "eventId": "DE2EC5FA-AF0A-4D59-85D2-677C66A6BC12", "preemptionTime": "", "eventType": "Preempt", "resourceGroup": "RG-TEST", "subscription": "azure-sub-id", "vmName": "anivmnew" }, "id": "85c058fc-4f2e-49ec-a027-6fcca60f7935", "subject": "/subscriptions/azure-sub-id/resourceGroups/RG-TEST/providers/Microsoft.Compute/virtualMachines/anivmnew", "eventType": "SpotVM.Preemption", "dataVersion": "1.0" } ] 2025-03-19 20:05:01 - Event Grid response: 2025-03-19 20:05:01 - Taking actions to prepare for shutdown... C) Configuring event subscribers Now that your Spot VMs are sending eviction events to Event Grid, set up subscribers to take action when these events occur. For example sending event to service bus queue: Conclusion By implementing this solution, you've created a reliable way to detect and respond to Spot VM evictions. This approach gives your applications precious time to react to evictions, significantly improving reliability while still benefiting from the cost savings of Spot VMs. While Azure does provide resource-level events through system topics, they simply don't provide the reliability, timing, and clarity needed for mission-critical workloads running on Spot VMs. The combination of the Azure Instance Metadata Service Scheduled Events API and custom Event Grid topics creates a powerful pattern for building resilient, event-driven architectures. This approach ensures you're getting the most accurate and timely notifications possible, giving your applications the best chance to gracefully handle Spot VM evictions while enjoying the substantial cost benefits that Spot VMs offer. Disclaimer The sample scripts provided in this article are provided AS IS without warranty of any kind. The author is not responsible for any issues, damages, or problems that may arise from using these scripts. Users should thoroughly test any implementation in their environment before deploying to production. Azure services and APIs may change over time, which could affect the functionality of the provided scripts. Always refer to the latest Azure documentation for the most up-to-date information. Thanks for reading this blog! I hope you've found this approach to handling Spot VM evictions helpful
Azure VM - RemoteApps Mouse Focus Issue
Seeing an issue this morning on multiple VMs and hostpools where Remoteapps have not handling Window Focus correctly. Situation: A maximized app and a secondary app smaller and on the same screen. If attempting to click or use the smaller app. The maximized app is receiving the mouse clicks. In some situations it will even pull to the front covering the smaller app. The only way to interact with the smaller window is to minimize or move the larger window out of the way. Both applications in the scenario are remote apps. Naturally anything running natively off the client PC is unaffected. No updates to the client PC. Trying to identify what would cause this behavior overnight.
RD Client Android Fail...
Serious question, why is RD Client for Android the worst RD client? It does not work with Samsung DEX, sure it runs and connects, but fails at basic usability, like recognizing the actual screen size when windowed. Also, dosn't resize properly. Pretty basic stuff, nothing remotely new here. The Citrix and VM Ware clients just work, and work well. Microsoft and Samsung have a partnership in the phone space which makes this even more absurd. So, what's the deal? Laziness, incompetence, just don't care anymore? Will this ever be fixed? This makes me worry about every other product.
mstsc : The term 'mstsc' is not recognized as the name of a cmdlet, function, script file.
I am using mstsc /v:publicIpAddress command to create remote desktop connection from local computer. but getting this error mstsc: The term 'mstsc' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. how to create remote desktop connection using CLI/powershell?
A3- standard VM pricing
Hello, I want to find out the pricing for A3 Standard VM running RHEL in "US West 2 " in Azure portal . I cannot locate anywhere . Only in the calculator when I input all the details , I can find the price . Anybody can give some input here? Azure portal pricing page gives only pricing for A3 - Basic .. This gives an impression that A3-standard does not exit . However in the calculator, you can input A3-standard .. How come this discrepancy ?? Thanks, Chitta
Azure Windows 10 vm available for non-msdn user?
Hi all, We are using Pay-As-You-Go subscription as far as i know windows 10 is not available non-msdn But now i am little confused about the product https://azuremarketplace.microsoft.com/en-us/marketplace/apps/Microsoft.Windows10RS3Prox64-ARM?tab=Overview in azure market place Can anyone please explain to me this product also belongs to msdn subscription or its actually available for PAYG subscription? Thanks Nihal
Sql Server Analysis Services are getting stopped in Azure VM every day
Hi All, Currently I came across with a weird type of issue which I am not sure whether I missed any step while setting up the Azure VM. Problem: I installed Sql Server Analysis Services in Azure VM and Analysis Services is getting stopped each day....!!!!! If someone came with the same kind of issue earlier kindly help. Thanks in Advance.
