A guide to Azure Storage and Virtual Machines cost optimization

In this blog, we will explore various techniques to optimize costs for Azure storage accounts and virtual machines, while also highlighting some important caveats to keep in mind. 

Where do we begin? 

Before diving into specific cost optimization strategies, it's essential to understand where your money is being spent. Microsoft Cost Management is a powerful tool that provides comprehensive insights into your cloud spending. By leveraging this tool, you can identify cost drivers, monitor spending patterns, and make informed decisions to optimize your expenses. 

Let's take a closer look at the Cost Management graph below to see how we can break down and analyse our expenses. 

 

The graph above shows the daily costs incurred by different services over a period of time. The x-axis represents the dates, while the y-axis represents the cost in USD ($). Different colors represent various services such as Virtual Machines, SQL Databases, Storage, etc. 

By analyzing this data, you can identify which services are driving your costs and take appropriate actions to optimize them. This foundational step is crucial for any cost optimization strategy and ensures that you are making data-driven decisions to maximize your cloud investments. 

Optimizing costs for storage accounts 

When it comes to Azure storage accounts, understanding where most of your money is being spent is crucial for effective cost optimization. Azure offers various storage options, each with its own pricing model and use cases. By analyzing your storage usage and implementing cost-saving strategies, you can significantly reduce your expenses.  

Again, Azure Cost Management is our savior :) you can find the cost breakdown of a storage account per meter, which will give us a better view of where to start our optimization from. 

 

Also, breakdown view of different tiers

 

Let's dive deeper...

Major cost drivers

Let's understand the major cost drivers and areas for optimization

• Storage tiers: Azure provides different storage tiers, including Hot, Cool, Cold, and Archive. The Hot tier for frequently accessed data and expensive. The Cool and Archive tiers are more cost-effective for infrequently accessed data. 

• Storage transactions: Every operation on your storage account, such as read, write, and delete, incurs a transaction cost. High transaction volumes can lead to significant expenses. 

• Data redundancy options: Azure offers several redundancy options, such as Locally Redundant Storage (LRS), Zone-Redundant Storage (ZRS), Geo-Redundant Storage (GRS), and Read-Access Geo-Redundant Storage (RA-GRS). Higher redundancy levels provide better data protection but come at a higher cost 

• Managed disks: Azure managed disks are block-level storage volumes that are managed by Azure and used with Azure Virtual Machines (VMs). They are designed to provide high durability, availability, and scalability for your applications. Not using appropriate disk type will lead to higher cost

Let's take a closer look at the optimization approach for each cost drivers including caveats and considerations.  

Storage tier recommendations: Hot to cold tier 

First analyse the read/write transactions & data storage utilization to understand the frequency of read/write transactions and the amount of data stored

• Leverage Cold tier to lower cost when there is less read and write operations with more data storage cost 

• For instance, moving data storage from the hot tier, which costs $0.195 per GB for the first 50TB/month, to the cold tier, which costs $0.0045 per GB for the first 50TB/month, can result in significant cost savings. Specifically, this transition can save approximately 97.69% on storage costs 

• In summary, less storage cost but more transaction costs 

Caveats & considerations: Hot to cold tier 

When you change the storage tier from hot to cold, azure charges for write operations. Specifically, the cost of tiering down from hot to cool can be estimated by the number of write operations required, which is calculated per 10,000 operations. You will need to pay the cost due to write operations when migrating their storage account from the hot tier to the cold tier. 

This is a one-time cost incurred during the migration process. After the initial migration, the ongoing storage costs in the cold tier should be lower compared to the hot tier, leading to long-term savings 

Benefits:

The long-term savings will be realized over time as the daily storage costs in the cold tier are lower compared to the hot tier.

Storage transactions recommendations: V2 to V1 

Please be mindful the use of V1 storage account from V2 should be considered very carefully, though it's not an ideal approach, it can be a viable option to explore.

• Leverage V1 type when you pay more for transactions with less data storage cost 

• This recommendation is particularly beneficial when transactions costs are a significant portion of the overall storage expenses 

• This option can be leveraged when you don’t leverage most of V2 storage features, migrating to V1 would trigger significant saving 

• In summary, less transaction cost but more data storage cost 

Caveats & considerations: V2 to V1 

• Transition from V2 to V1 is not a straightforward approach, it requires the creation of a new storage account with V1 type. 

• As of today, you cannot create a V1 account from azure portal, this can be created via IAC (bicep, powershell, terraform...). Post the account creation, you must move all the historical data to the new account. 

• Migrating historical data from existing accounts will incur one time data migration costs as per V1 write transaction cost. 

• General-purpose v1 accounts do not provide access to Cool or Archive storage.  

• Since V1 storage accounts do not support Lifecycle Management policies. To manage data retention, you must create a solution to archive the data that has not been accessed in a while. 

Benefits: 

The long-term savings will be realized over time as the daily transaction costs in v1 are lower compared to v2. 

Data redundancy recommendations: Choosing the right option 

It's crucial to balance the need for redundancy with budget constraints. 

• While higher redundancy options like Geo-Redundant Storage (GRS) provide better data protection, they come at a higher cost. 

• Higher redundancy options may introduce latency due to data replication across regions. This can impact the performance of applications that require low-latency access to data 

By carefully evaluating the redundancy requirements a substantial savings can be seen. 

Managed disks recommendations:

Managed disks can significantly impact your overall storage costs if not planned and utilized properly. Premium disks are designed for high-performance and low-latency workloads. However, due to their higher cost compared to standard disks, it's essential to use them judiciously and ensure they are allocated based on actual performance needs 

• Analyse the usage of premium disk on their virtual machines (VMs). Specifically, focus on VMs that are primarily used for read-only caching. Understand the performance requirements and whether the premium disks can be changed to standard disk as you don't need intensive write operation to perform 

• Consider moving premium disks to standard disks in lower environments, such as development and testing environments, if possible. 

• Consider using ephemeral disks for AKS node pools. Ephemeral disks are temporary storage that is directly attached to the VM and provides high performance at a lower cost. If you don't need data persistence for your stateless workloads, ephemeral disks would be right choice. 

• Finally, identity & remove orphan disks – disks that are not attached to any VMs. These disks can incur unnecessary costs. 

• Orphan disks can be identified by many ways you can have a script, and other option would be using an azure workbook that can be handy to find orphan resources. dolevshor/azure-orphan-resources: Centralize orphan resources in Azure environments

Caveats & considerations: Premium to standard 

• Premium disks offer higher IOPS (Input/Output Operations Per Second) and lower latency compared to standard disks. Moving to standard disks may result in reduced performance, which could affect applications that require high-speed data access 

• Make sure to analyze your usage patterns and see if moving from premium to standard disks is the right choice for your specific use case 

Optimizing costs for Azure virtual machines

Managing virtual machine costs effectively is crucial for profitability and sustainable growth. Over-provisioned resources often lead to unnecessary expenses, but optimizing resource usage and reducing unnecessary VMs can result in substantial savings and positive environmental impacts. 

 

 

Above image is a representation of different SKU used and the wasted cost on that SKU’s. Segregation of virtual machines are on SKU basis and each SKU cost is the sum of multiple VM’s in a particular SKU. 

Major cost drivers 

Let us understand the major cost drivers and areas for optimization 

Stock keeping unit (SKU):  

The selection of the right Stock Keeping Unit (SKU) is a critical factor in optimizing costs. Different SKU types serve various needs: 

• B Series: Ideal for small, non-critical environments with low resource consumption. 

• F Series: Suitable for CPU-intensive workloads requiring high computational power. 

• E Series: Best for memory-intensive applications needing substantial RAM. 

• D Series: A general-purpose SKU that is often the default choice but might not always be the most cost-efficient. 

Avoiding the default selection of D Series and choosing the appropriate SKU based on specific application requirements can significantly reduce costs and improve efficiency. 

 

 

 

This image is representing the CPU utilization of Different virtual machines under a particular Subscription. By analyzing the usage, you can define the appropriate SKU for your Virtual Machine. 

Recommendations: - Choosing correct SKU series: 

First, start monitoring CPU and memory utilization of virtual machines to identify the exact load on virtual machines. 

• Create a workbook to monitor the utilization or use if there are any tools to monitor the CPU and memory utilization. 

• if the workload is for a small environment and not critical then please use B series. 

• if it is more CPU intensive then choose F series and if it is memory intensive then choose E series. 

• Basis on the load and cost please choose the best SKU, also use the latest version of SKU. 

• For memory intensive Virtual machines i.e. Standard_D16_v3 will cost around $1.504 per hour and moving it to E series i.e. Standard_E8_v5 will cost around $0.872 per hour may lead to save ~ 60 % of cost. 

• Using the latest version of SKU will also lead to save cost i.e. Standard_D16_v3 will cost around $1.672 per hour and using the latest version i.e. Standard_D16_v5 will cost around $1.632 per hour.  

Databricks VM SKU recommendations

• Balance cost efficiency and performance when selecting Databricks VM SKUs. 

• Monitor CPU and memory utilization to choose the appropriate series (B for low cost, F for CPU-intensive, E for memory-intensive tasks). 

• Consider upgrading to the latest SKU versions for potential cost savings. 

• Use automation on data bricks VM to save extensive cost. 

• Use reserved instances for 1-3 years to save ~ 60 % cost. 

Reservations for virtual machines

• Reserving virtual machines can help optimize resource usage and reduce operational costs. 

• Committing to a reservation plan allows businesses to take advantage of significant discounts compared to pay-as-you-go pricing. 

• This approach is particularly beneficial for workloads with predictable and consistent usage patterns. 

Auto-shutdown of virtual machines 

Auto-shutdown of virtual machines saves energy costs and frees resources, enhancing both performance and sustainability. It also limits unauthorized access, aiding in compliance with data privacy regulations. 

Shutdown and Deallocation of Virtual Machines: - 

Virtual machines not used for long time can be deallocated to save the extra cost of resources/ components attached to the virtual machine.  

Virtual machines used for a particular time can be auto shutdown to save the unwanted charges. 

Orphaned disks (unattached disks) 

Orphaned Disks: Storage disks that remain unattached to any virtual machine (VM) within the cloud environment, leading to unnecessary costs and resource inefficiencies. 

Identification and Management 

• Use tools like Azure workbooks to detect unattached disks. 

• Use resource graph explorer query to get the details of orphaned disks. 

• Implement policies and alerts for prompt notifications. 

• Establish routine clean-up practices to either reattach or delete orphaned disks. 

Efficient management minimizes costs and optimizes resource utilization. 

Resource Graph queries

Below are the resource graph queries that were particularly handy for the storage & VM findings.

 

To find premium disk with read-only caching: 

resources 

| where type == "microsoft.compute/virtualmachines" 

| join kind=inner ( 

    ResourceContainers 

    | where type == "microsoft.resources/subscriptions" 

    | project subscriptionId, subscriptionName = name 

) on subscriptionId 

| mv-expand dataDisks = properties.storageProfile.dataDisks 

| extend osDisk = properties.storageProfile.osDisk 

| project resourceGroup, subscriptionName, location, diskName = dataDisks.name, caching = dataDisks.caching, diskType = "DataDisk", sku = dataDisks.managedDisk.storageAccountType 

| union ( 

    resources 

    | where type == "microsoft.compute/virtualmachines" 

    | join kind=inner ( 

    ResourceContainers 

    | where type == "microsoft.resources/subscriptions" 

    | project subscriptionId, subscriptionName = name 

    ) on subscriptionId     

    | project resourceGroup, subscriptionName, location, diskName = properties.storageProfile.osDisk.name, caching = properties.storageProfile.osDisk.caching, diskType = "OSDisk", sku = properties.storageProfile.osDisk.managedDisk.storageAccountType 

) 

| where caching == "ReadOnly" 

 

To find Storage V2 account with Hot tier for cold migration: 

Resources 

| where type == "microsoft.storage/storageaccounts" 

| join kind=inner ( 

    ResourceContainers 

    | where type == "microsoft.resources/subscriptions" 

    | project subscriptionId, subscriptionName = name 

) on subscriptionId 

| project subscriptionName, name, kind, location, resourceGroup, sku.tier, Tier = properties.accessTier, replication = sku.name 

| where kind == 'StorageV2' and Tier == 'Hot' 

 

To find ephemeral & managed disks attached to VMSS (AKS Nodepool): 

resources 

| where type == "microsoft.compute/virtualmachinescalesets" 

| join kind=inner ( 

    ResourceContainers 

    | where type == "microsoft.resources/subscriptions" 

    | project subscriptionId, subscriptionName = name 

) on subscriptionId 

|project subscriptionId, subscriptionName, name, instancesCount = tolong(sku.capacity), location, 

        sku = sku.name, 

        diskSize = properties.virtualMachineProfile.storageProfile.osDisk.diskSizeGB,  

        storageType = properties.virtualMachineProfile.storageProfile.osDisk.managedDisk.storageAccountType,  

        caching = properties.virtualMachineProfile.storageProfile.osDisk.caching, 

        diskType = iif(isnull(properties.virtualMachineProfile.storageProfile.osDisk.diffDiskSettings), "Managed", "Ephemeral") 

| where storageType contains "Premium" 

| where name contains "aks" 

 

To find the orphaned disks: 

Resources | where type == "microsoft.compute/disks" 

| join kind=inner ( 

    ResourceContainers 

    | where type == "microsoft.resources/subscriptions" 

    | project subscriptionId, subscriptionName = name 

) on subscriptionId 

 | where isnull(properties.diskState) or properties.diskState == "Unattached" 

 | project name, id, location, resourceGroup, properties, subscriptionId, subscriptionName 

 

To find the details of VM: 

Resources 

| where type == "microsoft.compute/virtualmachines" 

| join kind=inner ( 

    ResourceContainers 

    | where type == "microsoft.resources/subscriptions" 

    | project subscriptionId, subscriptionName = name 

) on subscriptionId 

| project vmName = name, vmId = id, subscriptionId, subscriptionName, location, resourceGroup, sku = properties.hardwareProfile.vmSize, osType = properties.storageProfile.osDisk.osType 

 

 

To find the VM details not containing Data bricks VM: 

Resources 

| where type == "microsoft.compute/virtualmachines" 

| join kind=inner ( 

    ResourceContainers 

    | where type == "microsoft.resources/subscriptions" 

    | project subscriptionId, subscriptionName = name 

) on subscriptionId 

| project vmName = name, vmId = id, subscriptionId, subscriptionName, location, resourceGroup, sku = properties.hardwareProfile.vmSize, osType = properties.storageProfile.osDisk.osType 

| where resourceGroup !contains "databricks" and resourceGroup !contains "adb" 

References: 

• Azure workbook to find orphan resources - dolevshor/azure-orphan-resources: Centralize orphan resources in Azure environments 

• Azure workbook to find the CPU utilization - Azure Workbooks overview - Azure Monitor | Microsoft Learn