Whether you consider yourself a FinOps practitioner, someone who’s enthusiastic about driving cloud efficiency and maximizing the value you get from the cloud or were just asked to look at ways to reduce cost, the FinOps toolkit has something for you. This month, we have a few highly anticipated releases: support for large datasets at scale with Azure Data Explorer, EA and MCA cost savings, private endpoints, and infrastructure encryption in FinOps hubs; a workaround for the double-counted savings plan usage in Power BI; and many more small updates and improvements across the board. There’s a lot of detail behind our Azure Data Explorer support. Read on for details!

New to FinOps toolkit?

In case you haven’t heard, the FinOps toolkit is an open-source collection of tools and resources that help you learn, adopt, and implement FinOps in the Microsoft Cloud. The foundation of the toolkit is the Implementing FinOps guide that helps you get started with FinOps whether you’re using native tools in the Azure portal, looking for ways to automate and extend those tools, or if you’re looking to build your own FinOps tools and reports. To learn more about the toolkit, how to provide feedback, or how to contribute, see the FinOps toolkit site.

Private endpoints for FinOps hubs

FinOps hubs come with a few big improvements in 0.7. First up is support for private endpoints for enhanced security. With this update, FinOps hubs are now deployed within a dedicated virtual network (VNet). Most resources are deployed within the FinOps hub VNet to ensure data processing happens within its security boundary. The main exception is the integrated runtime used by Azure Data Factory when private endpoints are not enabled.

When deploying a new hub instance, you’ll see a few new options in the deployment form (or template parameters, if deploying programmatically). You’ll find private endpoints on the Advanced tab where you can set Access to either Public or Private, depending on your needs.

• Public enables public access to ADLS and Data Explorer (if deployed) via their respective firewalls. Azure Data Factory is configured to use the public integration runtime, which helps reduce costs.
• Private disables public access to ADLS and Data Explorer (if deployed) and deploys a managed VNet and integration runtime for Azure Data Factory. The managed integration runtime ensures data processing happens within the VNet but does incur additional cost.

You can also customize the VNet address prefix from the “10.20.30.0/26” default value but note that “/26” is required. Internally, FinOps hubs uses the /28 subnet for private endpoints, another /28 subnet for temporary deployment scripts (container instances), and a /27 subnet for Azure Data Explorer, if enabled.

Public access (via the service’s firewall) is enabled by default to support existing deployments. While this traffic is still secured with least-privileged role-based access control (RBAC) permissions, Microsoft recommends private endpoints for organizations looking for the additional layer of security. If you have an existing network, you can leverage VNet peering to ensure open communication between your existing VNet and the FinOps hub VNet.

When using private endpoints, please ensure you consult the pricing calculator to fully understand the impact on pricing. There will be additional cost for data transferred through the Azure Data Factory managed VNet, peering with an existing VNet, and Power BI licenses. Costs with private endpoints can be significantly higher than the historical $5 per million per month for storage and Azure Data Factory. This will also impact any custom pipelines reading from or writing to the hub storage account, such as ingesting non-Azure FOCUS cost data.

We hope this additional layer of security makes it easier to leverage FinOps hubs within your organization. We’ve heard from many organizations who have been waiting for this. And on behalf of the team, I also want to extend a big thank you to Patric K (@patkje75) for kicking this all off with his PR to add private endpoints support in hubs. It went thru several iterations, but it’s great to finally see it come to life. Thanks, Patric!

Unlocking scalable analytics for FinOps hubs with Azure Data Explorer

The FinOps hubs vision was founded on three principles to build an open, extensible, and scalable solution based on community standards and best practices. FinOps hubs started as an open solution. Then, we adopted community standards and best practices by introducing the FinOps Open Cost and Usage Specification (FOCUS) and aligning to the FinOps Framework in 0.2. Now, in 0.7, we’re adding the foundation for scalable analytics with Azure Data Explorer, setting the stage for a broad spectrum of FinOps capabilities yet to come!

There’s no way to cover everything Data Explorer unblocks in a single blog post, so I’ll focus on a few of the key improvements we’ve heard the most about over the past year.

First and foremost, Data Explorer comes with a powerful, high-performance query language that resolves Power BI data refresh limitations some organizations are seeing today. We’re currently hearing that FinOps toolkit storage-based reports are capping out at ~$2 million in monthly spend. With Data Explorer, we’re able to jump past this limit, seeing some queries go from hours to minutes or even seconds. We haven’t identified the exact cap but will continue to tune queries, reports, and the underlying database to maximize coverage as we learn and grow.

To support this scale, we’ve introduced a new Daily or Monthly granularity parameter in Power BI. This option is only available for reports designed for Data Explorer, but it allows organizations to view daily data for a shorter period (e.g., 1 month) while also supporting longer-term reporting at monthly granularity.

Unlike storage, KQL queries have row and size limits which require a much different approach to querying and reporting on data in Power BI. To support the new, faster query language, we created a new set of Power BI reports. In 0.7, you can download a set of demo reports with sample data (PowerBI-demo.zip) or you can download a set of templates that work against storage (PowerBI-storage.zip) or KQL (PowerBI-kql.zip).

Taking that even further, historically, you haven’t had many options for how to query data in storage. While Power BI is great, creating more advanced queries with Power Query can be incredibly challenging and can dramatically slow down data refresh times – which is exactly what we’ve witnessed over the past year with FinOps toolkit reports. But with Data Explorer, you can use KQL to run advanced queries on your cost data in seconds, answering questions you may not have even realized was possible. I’m looking forward to sharing many of these in new dashboards, workbooks, and the FinOps best practice library we launched in September.

One example of how we’ve been able to leverage this power is that FinOps hubs 0.7 with Data Explorer now shows missing list and contracted costs enabling you to calculate total cost savings for EA and MCA accounts! By exporting both your billing account/profile price sheet as well as FOCUS cost data, FinOps hubs looks up the missing prices, calculates costs, and offers queries that provide a more complete savings calculation.

Of course, this is just one of the many data normalization and cleansing benefits you’ll find in FinOps hubs with Data Explorer. To learn more about how we’re augmenting and improving your data, see About Data Explorer ingestion. If you’re neck-deep in cost and price data and curious about what’s next in this area as we identify and implement additional data quality improvements, check out the data ingestion transform backlog (issue #1111). Please leave comments to help us identify the best areas to invest in to help you accomplish your goals.

To deploy FinOps hubs with Data Explorer, simply specify the cluster name as part of the deployment form and we’ll set that up as part of the deployment and begin ingesting data directly from Azure Data Factory.

You can also indicate what SKU to use with Data Explorer, which enables resiliency and more scalable data ingestion and querying when under load. We expect most organizations to be able to leverage the defaults.

Lastly, you can also indicate data retention settings in Data Explorer. Raw data generally does not require retention, but may be useful for troubleshooting – especially when identifying data lineage of data quality issues.

We’re excited to hear how this works for you. We have many optimizations planned on top of this, so it’s a perfect time to try it out and let us know how we can improve the tool for you.

Double counting savings plan purchases in Power BI reports

One issue that has been reported more over the past month or so is the inconsistent EffectiveCost number in FOCUS data compared to amortized cost in the Azure portal. The reason for this is due to a bug in Cost Management where savings plan purchases are tracked as amortized (effective) cost. While we wait for the underlying issue to get resolved in Cost Management data, we’ve updated FinOps hubs and Power BI reports in 0.7 to account for this by removing the invalid effective cost values. You’ll see this in storage-based reports using Cost Management exports or FinOps hubs with storage as well as KQL queries and reports when using FinOps hubs with Data Explorer.

As always, let us know if you spot any additional data quality issues. We’re happy to help investigate and see how we can work around them to streamline your FinOps reporting and optimizations efforts.

Other new and noteworthy updates

Many small improvements and bug fixes go into each release, so covering everything in detail can be a lot to take in. But I do want to call out a few other small things that you may be interested in.

In the Implementing FinOps guide:

• Added enterprise application patterns to the Architecting for cloud capability.
• Updated FOCUS conversion instructions based on community feedback.

In FinOps hubs:

• Changed the storage ingestion path from “{export-type}/{yyyy}/{MM}/{scope}” to “{dataset}/{yyyy}/{MM}/{scope}” for more flexible data ingestion. This is a breaking change
• Added support for FOCUS 1.0r2 exports.
• Added support for storage account infrastructure encryption.
• Renamed the msexports_FileAdded trigger to msexports_ManifestAdded.
• Published the schema file for the hub settings.json file to support autocomplete.

In Power BI reports:

• Added partial support for OneLake URLs.
• Consolidated Hub Storage URL and Export Storage URL parameters.
• Renamed x_Dataset* columns to x_Source*.
• Renamed x_AccountType column to x_BillingAccountAgreement.
• Updated supported spend estimates in documentation to reflect reported limitations of ~$2 million per month.

In FinOps workbooks:

• On the Optimization workbook Storage tab, included the RSVaultBackup tag in the list of non-idle disks.
• On the Optimization workbook Azure reservations tab, fixed a configuration issue which was limiting results to 100 rows.
• On the Optimization workbook Compute tab, fixed incorrect virtual machine processor in processors query.
• On the Optimization workbook Database tab, removed the idle SQL database query.

In Azure optimization engine:

• Fixed exports ingestion issues in cases where exports come with empty lines (see issue #998).
• Fixed missing columns in EA savings plans exports (see issue #1026).

In open data:

• Added 80 new and updated 26 existing resource types.
• Added 3 new resource type to service mappings.

What’s next

While this was a big release with a lot of great stuff, it sometimes feels like we’ve added more to our backlog than taken away from it. It’s truly exciting to think about the direction that we’re headed and how close we’re getting to some of the bigger challenges we want to address.

That said, we also know there’s room for tuning what we’ve released in 0.7. Here are a few of the things we’re looking at in the coming months:

• FinOps hubs will continue to invest in data quality and completeness to enable more FinOps scenarios.
• Power BI reports will continue to be optimized. Expect storage reports to be simplified and for data in KQL reports to be restructured to support even more data at scale.
• FinOps workbooks will continue to get recurring updates, expand to more FinOps capabilities, and add cost from FinOps hubs.
• Azure Optimization Engine will continue to receive small updates as we begin to bring some capabilities into FinOps hubs in upcoming releases.
• Each release, we’ll try to pick at least one of the highest voted issues (based on 👍 votes) to continue to evolve based on your feedback, so keep the feedback coming!

To learn more, check out the FinOps toolkit roadmap, and please let us know if there’s anything you’d like to see in a future release. Whether you’re using native products, automating and extending those products, or using custom solutions, we’re here to help make FinOps easier to adopt and implement.