All Key Vaults Are Critical, But Some Are More Critical Than Others: Finding the Crown Jewels

Introduction 

A critical asset is one of substantial value, whose compromise or disruption would result in significant adverse effects on the organization. This definition lays the foundation for understanding why Azure Key Vaults often fall into this category. 

Azure Key Vaults are integral to cloud environments as they manage sensitive data like cryptographic keys, passwords, and certificates. Their frequent use in securing applications, managing secrets, and enabling secure operations makes them highly valuable. Given this importance, identifying which Key Vaults are critical becomes essential.

Approach 

Our approach to identifying critical Key Vaults is based on operational activity. We classify Key Vaults using the top n percentile of operations within each tenant, ensuring that only the most active and essential Key Vaults are flagged as critical. This approach provides a fair evaluation across varying tenant sizes and ensures that thresholds dynamically adjust with data size and distribution, making the classification resilient to outliers and representative of actual operational importance. 

Why Focus on Key Vaults with High Operation Counts? 

Increased Usage Indicates High Dependency: 

A high volume of operations suggests that the Key Vault is heavily utilized, meaning it plays a central role in the security and operational processes within the environment. For example, it might be frequently accessed to retrieve secrets, keys, or certificates, which are essential for the functioning of various applications and services. 

Sensitive Data Storage: 

Key Vaults typically store sensitive data, such as cryptographic keys, passwords, and other secrets. A Key Vault with many operations is likely to store and manage a significant amount of this sensitive data, making it a high-value target for potential attacks. 

Operational Impact: 

If a heavily used Key Vault were compromised or became unavailable, it could disrupt multiple critical processes across the organization. This could include application outages, security breaches, or other operational failures, making the Key Vault critical to overall business continuity. 

Security Implications: 

Frequent access to a Key Vault might indicate its role in automated processes or scripts that require secure handling of credentials and keys. The more a Key Vault is accessed, the higher the potential risk if its security is breached, hence making it essential to protect and monitor it closely. 

Benefits of Using Percentiles in Criticality Classification 

In critical asset classification, the use of percentiles offers several distinct advantages over percentage-based methods: 

Resilience to Outliers:

Percentiles rank Key Vaults without being influenced by extreme values. For instance, even if one Key Vault has an unusually high operation count, the percentile method ensures that the classification threshold remains stable. 

Dynamic Adaptation to Dataset Size:

As the number of Key Vaults grows, percentile thresholds adjust dynamically, maintaining consistency and accuracy over time. 

Fair Evaluation Across Tenants:

Different tenants have varying numbers of Key Vaults. Percentiles allow for a fair assessment by ensuring that each tenant’s Key Vaults are evaluated within that tenant’s dataset. This means that even smaller tenants with fewer Key Vaults can have their most active Key Vaults identified as critical without being overshadowed by the larger operation counts of bigger tenants. Percentiles rank within each tenant individually, making the classification equitable across different scales. 

Mathematical Rigor:

Percentiles provide a statistically sound method for ranking Key Vaults, offering a reliable framework for criticality classification. 

Operational Relevance:

By using percentiles, the classification highlights Key Vaults that are truly operationally significant within their own environment, enhancing security monitoring and response efforts. 

This approach ensures that critical assets are identified accurately, without the distortions caused by outliers, dataset size, or operational scale variations, making it ideal for cloud environments. 

Findings from Research 

• Overall Critical Assets: Around 0.5% of total KVs were identified as critical

• Tenant-wise Analysis: Percentile thresholds adjusted dynamically across tenant sizes. 

• Large tenants saw a minimal increase in critical assets, validating accuracy. 

• Smaller tenants benefited from nuanced classification. 

Percentile-based classification ensures that Key Vaults with relatively high operation counts are identified, regardless of tenant size, providing a balanced approach. 

 

 

Figure 1: Tenant-wise Analysis  

 

Finding the Optimal Percentile Threshold 

The reverse elbow curve method is a data-driven approach to determine the optimal percentile threshold. Figure 2 illustrates this concept by plotting the percentage of Key Vaults classified as critical against various percentile values. As the percentile value increases from 90 to 99, the percentage of critical Key Vaults decreases, forming a clear reverse elbow shape. 

In this graph, the curve starts to flatten around the 95th percentile, marked as the 'Optimal Percentile Threshold.' This point represents where the rate of decrease in critical Key Vaults slows down significantly. Selecting this threshold ensures that we capture the most critical Key Vaults without unnecessarily including too many lower-priority assets. Before this point, too many Key Vaults are classified as critical, while after this point, too few Key Vaults are included. 

 

Figure 2: Identifying the optimal percentile threshold 

This visual example demonstrates why the reverse elbow curve method is essential for balancing coverage and precision in critical asset classification, ensuring that the most operationally significant Key Vaults are identified efficiently. 

Conclusion 

In conclusion, identifying critical Azure Key Vaults is essential for maintaining the security, availability, and operational integrity of cloud environments. By leveraging a percentile-based classification approach, we ensure that only the most active and essential Key Vaults are recognized as critical assets. The use of the reverse elbow curve method further strengthens this classification by selecting an optimal percentile threshold that balances coverage and precision. This methodology not only minimizes noise from less active Key Vaults but also ensures that highly utilized and sensitive Key Vaults receive the attention they deserve. As cloud operations continue to scale, such data-driven classification approaches are vital for effective security management and risk mitigation.