Azure Data Explorer (ADX) is commonly used for monitoring cloud resources and IoT devices performance and health. This is done by continuous collection of multiple metrics emitted by these sources, and on-going analysis of the collected data to detect anomalies. The analysis is applied over time series of the relevant metrics in order to locate significant deviations of the metrics values relative to their typical normal baseline pattern.
Univariate Anomaly Detection
ADX contains native support for detecting anomalies over multiple time series by using the function series_decompose_anomalies(). This function can analyze thousands of time series in seconds, enabling near real time monitoring solutions and workflows based on ADX. For further information on using this function take a look at Anomaly detection & forecasting in Azure Data Explorer. This function is based on univariate analysis, i.e. analyzing each metric independently (of other metrics) for anomalies. Univariate analysis is simpler, faster, and easily scalable and is applicable to most real-life scenarios; however, there are some cases where it might miss anomalies that can only be detected by analyzing multiple metrics at the same time.
Multivariate Anomaly Detection
Let’s see an example of missing anomalies when using univariate analysis. Suppose we monitor two car metrics: speed and engine rpm. Having 0 for specific metric is not anomalous – either the car doesn’t move, or its engine is off. But measuring speed of 40 km/hour with 0 rpm is definitely anomalous - the car might be sliding or pulled. To detect these types of anomalies we need to use multivariate analysis methods that jointly analyze time series of multiple metrics.
To understand the strength of multivariate analysis let’s try to detect anomalies of Microsoft stock over the last two years relative to SPY fund (that represents the top 500 US companies). First, let’s look at price charts of both securities:
We can see that for most of the trading days both securities’ prices change is in ±2% range, and the changes also correlate well. Now let’s look for multivariate anomalies using the elliptic envelope model:
The red dots are the anomalies. The orange circles that mark five extreme anomalies are intuitive to understand, as the daily change is above 6% or below -6%. But why is the black circled point on 27th April anomalous?
| mv-expand MSFT to typeof(real), SPY to typeof(real), anomalies to typeof(string)
| render scatterchart with(series=anomalies)
We can see that the orange anomalies are extreme for either MSFT or SPY or both, so they could have been detected by a univariate model, but the black one is normal for each security independently and can only be detected when jointly analyzing both securities by a multivariate model. Let’s look back on the time chart, zooming over that time window:
We can clearly see the anomalous behavior on 27th April - MSFT price increased by 1.65%, while SPY’s decreased by 2.01%.
Time series-based anomaly detection is a very powerful method for health monitoring of cloud resources and IoT devices. Many metrics are collected and can be analyzed for anomalies. ADX has built in capabilities for applying univariate anomaly detection for thousands of time series in near real time. Further, anomalies that were found on different metrics can be time based correlated to increase their score/confidence. In most cases this method is effective to detect issues; however, for some scenarios there might be a need for a true multivariate model, jointly analyzing multiple metrics. This can be achieved now in ADX using the new Python based multivariate anomaly detection functions. You are welcome to try these functions and share your feedback with us, as we continue improving and adding more multivariate capabilities.