Multivariate Anomaly Detection in Azure Data Explorer
Published Dec 04 2022 09:49 AM 2,849 Views
Microsoft

Introduction

Azure Data Explorer (ADX) is commonly used for monitoring cloud resources and IoT devices performance and health. This is done by continuous collection of multiple metrics emitted by these sources, and on-going analysis of the collected data to detect anomalies. The analysis is applied over time series of the relevant metrics in order to locate significant deviations of the metrics values relative to their typical normal baseline pattern.

 

Univariate Anomaly Detection

ADX contains native support for detecting anomalies over multiple time series by using the function series_decompose_anomalies(). This function can analyze thousands of time series in seconds, enabling near real time monitoring solutions and workflows based on ADX. For further information on using this function take a look at Anomaly detection & forecasting in Azure Data Explorer. This function is based on univariate analysis, i.e. analyzing each metric independently (of other metrics) for anomalies. Univariate analysis is simpler, faster, and easily scalable and is applicable to most real-life scenarios; however, there are some cases where it might miss anomalies that can only be detected by analyzing multiple metrics at the same time.

 

Multivariate Anomaly Detection

Let’s see an example of missing anomalies when using univariate analysis. Suppose we monitor two car metrics: speed and engine rpm. Having 0 for specific metric is not anomalous – either the car doesn’t move, or its engine is off. But measuring speed of 40 km/hour with 0 rpm is definitely anomalous - the car might be sliding or pulled. To detect these types of anomalies we need to use multivariate analysis methods that jointly analyze time series of multiple metrics.

We are happy to introduce three new UDFs (User Defined Functions) that apply different multivariate models on ADX time series. These functions are based on models from scikit-learn (the most common Python ML library), taking advantage of ADX capability to run inline Python as part of the KQL query. These are the new functions:

To understand the strength of multivariate analysis let’s try to detect anomalies of Microsoft stock over the last two years relative to SPY fund (that represents the top 500 US companies). First, let’s look at price charts of both securities:

 

TopStocksCleaned

| where Date >= datetime(2021-01-01)

| where Ticker in('SPY', 'MSFT')

| order by Ticker asc, Date asc

| render timechart with(ycolumns=AdjClose, ysplit=axes)

 

adieldar_0-1670173905809.png

 

We can see that MSFT quite correlates with SPY. Still, we would like to find anomalies in their mutual behaviors. For that purpose, let’s look at the daily % price changes:

 

TopStocksCleaned

| where Date >= datetime(2021-01-01)

| where Ticker in('SPY', 'MSFT')

| order by Ticker asc, Date asc

| extend Change = iff(Ticker == prev(Ticker), (AdjClose-prev(AdjClose))/prev(AdjClose)*100.0, 0.0)

| render timechart with(ycolumns=Change)

 

adieldar_1-1670173905817.png

 

We can see that for most of the trading days both securities’ prices change is in ±2% range, and the changes also correlate well. Now let’s look for multivariate anomalies using the elliptic envelope model:

 

TopStocksCleaned

| where Date >= datetime(2021-01-01)

| where Ticker in('SPY', 'MSFT')

| order by Ticker asc, Date asc

| extend Change = iff(Ticker == prev(Ticker), (AdjClose-prev(AdjClose))/prev(AdjClose)*100.0, 0.0)

| project Date, Ticker, Change

| evaluate pivot(Ticker, any(Change)) :(Date:datetime, MSFT:real, SPY:real)

| summarize Date=make_list(Date), MSFT=make_list(MSFT), SPY=make_list(SPY)

| extend anomalies=dynamic(null)

| invoke series_mv_ee_anomalies_fl(pack_array('MSFT', 'SPY'), 'anomalies', anomalies_pct=2)

| render anomalychart with(anomalycolumns=anomalies)

 

adieldar_2-1670173905828.png

 

The red dots are the anomalies. The orange circles that mark five extreme anomalies are intuitive to understand, as the daily change is above 6% or below -6%. But why is the black circled point on 27th April anomalous?

Let’s review it in a scatter chart:

 

TopStocksCleaned

| where Date >= datetime(2021-01-01)

| where Ticker in('SPY', 'MSFT')

| order by Ticker asc, Date asc

| extend Change = iff(Ticker == prev(Ticker), (AdjClose-prev(AdjClose))/prev(AdjClose)*100.0, 0.0)

| project Date, Ticker, Change

| evaluate pivot(Ticker, any(Change)) :(Date:datetime, MSFT:real, SPY:real)

| summarize Date=make_list(Date), MSFT=make_list(MSFT), SPY=make_list(SPY)

| extend anomalies=dynamic(null)

| invoke series_mv_ee_anomalies_fl(pack_array('MSFT', 'SPY'), 'anomalies', anomalies_pct=2)

| project MSFT, SPY, anomalies

| mv-expand MSFT to typeof(real), SPY to typeof(real), anomalies to typeof(string)

| render scatterchart with(series=anomalies)


adieldar_3-1670173905832.png

 

We can see that the orange anomalies are extreme for either MSFT or SPY or both, so they could have been detected by a univariate model, but the black one is normal for each security independently and can only be detected when jointly analyzing both securities by a multivariate model. Let’s look back on the time chart, zooming over that time window:

 

adieldar_4-1670173905856.png

 

We can clearly see the anomalous behavior on 27th April - MSFT price increased by 1.65%, while SPY’s decreased by 2.01%.

 

Summary

Time series-based anomaly detection is a very powerful method for health monitoring of cloud resources and IoT devices. Many metrics are collected and can be analyzed for anomalies. ADX has built in capabilities for applying univariate anomaly detection for thousands of time series in near real time. Further, anomalies that were found on different metrics can be time based correlated to increase their score/confidence. In most cases this method is effective to detect issues; however, for some scenarios there might be a need for a true multivariate model, jointly analyzing multiple metrics. This can be achieved now in ADX using the new Python based multivariate anomaly detection functions. You are welcome to try these functions and share your feedback with us, as we continue improving and adding more multivariate capabilities.

Co-Authors
Version history
Last update:
‎Dec 04 2022 10:10 AM
Updated by: