Home
Microsoft

Anomaly Detector  was on public preview on 3/26. We are pleased to see the adoption from a variety of customers across different industry verticals. We’d like to use this blog to provide additional details on Anomaly Detector and how to use it to achieve the best result with code walkthrough.

 

Overview

Anomaly detection aims to discover unexpected events or rare items in data. Accurate anomaly detection leads to prompt troubleshooting, which helps to avoid revenue loss and maintain brand reputation. At Microsoft, hundreds of teams rely on the technology we have built to monitor millions of metrics from Bing, Office, and Azure.

Anomaly Detector provides two APIs that detect anomalies automatically in time series with simple parameters, which require no machine learning background. It is designed for the scenarios of operational monitoring, business KPI monitoring, and IoT monitoring. Using Anomaly Detector APIs, you can infuse anomaly detection capabilities into your own platform and business process.

 

Who can use Anomaly Detector

Anomaly Detector is designed for anyone who can make a REST API call with script or code. It hides details of anomaly detection algorithms into only one main parameter to adjust the sensitivity of the detection. That makes it a perfect fit for integrating Microsoft anomaly detection capabilities into your business applications if you are tired of tuning the parameters for your AI algorithms or if you have no machine learning background.

 

Highlights

 

State of art anomaly detection system often uses a one size fit all approach. Which mean they apply some specific algorithm on all types of time series. Our learning is that each algorithm can handle some specific type of time series better. Our innovation is that we provide a generic framework to plug in different algorithm ensembles to handle a wide spectrum of different time series. In our different ensembles, we have following algorithms used.

  • Fourier Transformation
  • Extreme Studentized Deviate (ESD)
  • STL Decomposition 
  • Dynamic Threshold
  • Z-score detector
  • Some advanced algorithms to be disclosed pending paper publishing.

The following picture shows the algorithm selecting flow of Anomaly Detector. We will use another blog for more details on the algorithms.

alg_selection.png

 

Detecting all kinds of anomalies through one single endpoint

Besides spikes and dips, Anomaly Detector also detects many other kinds of anomalies, such as trend change and off-cycle softness, all in one single API endpoint.different_anomalies.png

Maintain simplicity outside: One parameter tuning

While running the Anomaly Detection service in Microsoft for several years, we have kept improving the one parameter strategy to make the tuning easier. The main parameter you need to customize is “Sensitivity”, which is from 1 to 99 to adjust the outcome to fit the scenario. Other advanced parameters are reserved for data scientists for further tuning.

 

Maintain sophistication inside: Selecting / Detecting / Filtering

To support one parameter tuning, we have put together a gallery of anomaly detection algorithms and designed a sophisticated strategy to leverage the collective power of machine learning to gain good precision and recall.

When we get the time series from an API request, the first thing is to extract features from the time series, for example continuity, mean, STD, trend, and period. With those features, the system selects the most appropriate algorithm.

It is not easy to find a good balance in recall and precision in one single detection without enough context. Anomaly Detector separates improving recall and precision into two steps. Firstly, it makes sure the recall is high. Then, it uses those features, which are extracted from the time series plus the sensitivity, which is the only parameter provided by the caller, to apply filtering on the results. This strategy has proved to provide better results.

detection_flow.png

 

Code walkthrough

The best way to show you the code and results is through Jupyter notebook, which you can see the code and visualization side by side. Let's go.

 

Simulate a streaming detection with "last" api

You can find the related sample data and Jupyter notebook here: https://github.com/Azure-Samples/anomalydetector/tree/master/ipython-notebook . Make sure you do read the readme here  first.

Let's start with Latest point detection with Anomaly Detector API.ipynb  notebook and walk through the code cell by cell.

By calling the API on your data's latest points, you can monitor your data as it's created.

The following example simulates using the Anomaly Detector API on streaming data. Sections of the example time series are sent to the API over multiple iterations, and the anomaly status of each section's last data point is saved. The data set used in this example has a pattern that repeats roughly every seven data points (the period in the request's JSON file), so for best results, the data set is sent in groups of 29 points (4 * period + an extra data point.).

Fill in the API access key and endpoint you received after creating your Anomaly Detector resource in the code block below. Refer here  on how to create AD resource.

The following request URLs must be combined with the appropriate endpoint for your subscription. For example:   

 https://westus2.api.cognitive.microsoft.com/anomalydetector/v1.0/timeseries/entire/detect

 

Batch detection

To detect anomalies throughout a batch of data points over a given time range, use the following request URI with your time series data:

/timeseries/entire/detect.

By sending your time series data at once, the API will generate a model using the entire series, and analyze each data point with it.

 

Streaming detection

To continuously detect anomalies on streaming data, use the following request URI with your latest data point:

/timeseries/last/detect'.

By sending new data points as you generate them, you can monitor your data in real time. A model will be generated with the data points you send, and the API will determine if the latest point in the time series is an anomaly.

# To start sending requests to the Anomaly Detector API, paste your subscription key below,
# and replace the endpoint variable with the endpoint for your region.
subscription_key = '' 
endpoint_latest = 'https://westus2.api.cognitive.microsoft.com/anomalydetector/v1.0/timeseries/last/detect'

You import necessary libraries for numerical calculation and visualization in the following code block.

import requests
import json
import pandas as pd
import numpy as np
from __future__ import print_function
import warnings
warnings.filterwarnings('ignore')

# Import library to display results
import matplotlib.pyplot as plt
%matplotlib inline


from bokeh.plotting import figure,output_notebook, show
from bokeh.palettes import Blues4
from bokeh.models import ColumnDataSource,Slider
import datetime
from bokeh.io import push_notebook
from dateutil import parser
from ipywidgets import interact, widgets, fixed
output_notebook()

The code block below shows you a detect function, which takes three parameters as inputs and returns the api call response in json data.

The three parameters are

  • The api endpoint you defined earlier
  • subscription_key you defined earlier
  • request_data the time series data you post to the api point

The return data is the meta data related to the anomaly detection results that you can find in the api definition.

def detect(endpoint, subscription_key, request_data):
    headers = {'Content-Type': 'application/json', 'Ocp-Apim-Subscription-Key': subscription_key}
    response = requests.post(endpoint, data=json.dumps(request_data), headers=headers)
    if response.status_code == 200:
        return json.loads(response.content.decode("utf-8"))
    else:
        print(response.status_code)
        raise Exception(response.text)

The code block below shows you a build_figure function, which takes three parameters as inputs and do the visualization upon extracted results, that is, sample data, anomalies, upper bound and lower bound, expected values...

def build_figure(result, sample_data, sensitivity):
    columns = {'expectedValues': result['expectedValues'], 'isAnomaly': result['isAnomaly'], 'isNegativeAnomaly': result['isNegativeAnomaly'],
              'isPositiveAnomaly': result['isPositiveAnomaly'], 'upperMargins': result['upperMargins'], 'lowerMargins': result['lowerMargins']
              , 'value': [x['value'] for x in sample_data['series']], 'timestamp': [parser.parse(x['timestamp']) for x in sample_data['series']]}
    response = pd.DataFrame(data=columns)
    values = response['value']
    label = response['timestamp']
    anomalies = []
    anomaly_labels = []
    index = 0
    anomaly_indexes = []
    p = figure(x_axis_type='datetime', title="Anomaly Detection Result ({0} Sensitivity)".format(sensitivity), width=800, height=600)
    for anom in response['isAnomaly']:
        if anom == True and (values[index] > response.iloc[index]['expectedValues'] + response.iloc[index]['upperMargins'] or 
                         values[index] < response.iloc[index]['expectedValues'] - response.iloc[index]['lowerMargins']):
            anomalies.append(values[index])
            anomaly_labels.append(label[index])
            anomaly_indexes.append(index)
        index = index+1
    upperband = response['expectedValues'] + response['upperMargins']
    lowerband = response['expectedValues'] -response['lowerMargins']
    band_x = np.append(label, label[::-1])
    band_y = np.append(lowerband, upperband[::-1])
    boundary = p.patch(band_x, band_y, color=Blues4[2], fill_alpha=0.5, line_width=1, legend='Boundary')
    p.line(label, values, legend='value', color="#2222aa", line_width=1)
    p.line(label, response['expectedValues'], legend='expectedValue',  line_width=1, line_dash="dotdash", line_color='olivedrab')
    anom_source = ColumnDataSource(dict(x=anomaly_labels, y=anomalies))
    anoms = p.circle('x', 'y', size=5, color='tomato', source=anom_source)
    p.legend.border_line_width = 1
    p.legend.background_fill_alpha  = 0.1
    show(p, notebook_handle=True)

The code block below shows you a detect_anomaly function, which takes sensitivity as input and repeatly call the detect function to simulate monitor streaming data as new data is created.

def detect_anomaly(sensitivity):
    sample_data = json.load(open('sample.json'))
    points = sample_data['series']
    skip_point = 29
    result = {'expectedValues': [None]*len(points), 'upperMargins': [None]*len(points), 
              'lowerMargins': [None]*len(points), 'isNegativeAnomaly': [False]*len(points), 
              'isPositiveAnomaly':[False]*len(points), 'isAnomaly': [False]*len(points)}
    anom_count = 0
    for i in range(skip_point, len(points)+1):
        single_sample_data = {}
        single_sample_data['series'] = points[i-29:i]
        single_sample_data['granularity'] = 'daily'
        single_sample_data['maxAnomalyRatio'] = 0.25
        single_sample_data['sensitivity'] = sensitivity
        single_point = detect(endpoint_latest, subscription_key, single_sample_data)
        if single_point['isAnomaly'] == True:
            anom_count = anom_count + 1

        result['expectedValues'][i-1] = single_point['expectedValue']
        result['upperMargins'][i-1] = single_point['upperMargin']
        result['lowerMargins'][i-1] = single_point['lowerMargin']
        result['isNegativeAnomaly'][i-1] = single_point['isNegativeAnomaly']
        result['isPositiveAnomaly'][i-1] = single_point['isPositiveAnomaly']
        result['isAnomaly'][i-1] = single_point['isAnomaly']
    
    build_figure(result, sample_data, sensitivity)

The following cells call the Anomaly Detector API with an example time series data set and different sensitivities for anomaly detection. Varying the sensitivity of the Anomaly Detector API can improve how well the response fits your data.


# 95 sensitvity
detect_anomaly(95)


# 85 sensitvity
detect_anomaly(85)

After all cells are executed, you would see following charts showing how sensitivity setting impacts the expected value bounds. The sensitivity is a way we allow users to tune the model with the understanding that different business or person could have different tolerance on even the same dataset. The higher the sensitivity, the narrower the band, as a result, you would get more anomalies.last_95.pnglast_85.png

 

 

 

 

Detect all anomalies within time series data in a batch with "entire" API

Let's switch to the Batch anomaly detection with Anomaly Detector API.ipynb  notebook and walk through the code cell by cell.

This notebook shows you how to send a batch anomaly detection request, and visualize the anomalies found throughout the example data set. The graph created at the end of this notebook will display the following items:

  • Anomalies found throughout the data set, highlighted.
  • The expected values versus the values contained in the data set.
  • Anomaly detection boundaries

You need to fill in the API access key and endpoint you received after creating your Anomaly Detector resource in the code block below.

# To start sending requests to the Anomaly Detector API, paste your subscription key below,
# and replace the endpoint variable with the endpoint for your region. 
subscription_key = '' 
endpoint = 'https://westus2.api.cognitive.microsoft.com/anomalydetector/v1.0/timeseries/entire/detect'

You import necessary libraries for numerical calculation and visualization in the following code block.

import requests
import json
import pandas as pd
import numpy as np
from __future__ import print_function
import warnings
warnings.filterwarnings('ignore')

# Import library to display results
import matplotlib.pyplot as plt
%matplotlib inline


from bokeh.plotting import figure,output_notebook, show
from bokeh.palettes import Blues4
from bokeh.models import ColumnDataSource,Slider
import datetime
from bokeh.io import push_notebook
from dateutil import parser
from ipywidgets import interact, widgets, fixed
output_notebook()

The code block below shows you a detect function, which takes three parameters as inputs and returns the api call response in json data.

The three parameters are

  • The api endpoint you defined earlier
  • subscription_key you defined earlier
  • request_data the time series data you post to the api point

The return data is the meta data related to the anomaly detection results that you can find in the api definition.

def detect(endpoint, subscription_key, request_data):
    headers = {'Content-Type': 'application/json', 'Ocp-Apim-Subscription-Key': subscription_key}
    response = requests.post(endpoint, data=json.dumps(request_data), headers=headers)
    if response.status_code == 200:
        return json.loads(response.content.decode("utf-8"))
    else:
        print(response.status_code)
        raise Exception(response.text)

The code block below shows you a build_figure function, which takes two parameters as inputs and do the visualization upon extracted results, that is, sample data, anomalies, upper bound and lower bound, expected values...

def build_figure(sample_data, sensitivity):
    sample_data['sensitivity'] = sensitivity
    result = detect(endpoint, subscription_key, sample_data)
    columns = {'expectedValues': result['expectedValues'], 'isAnomaly': result['isAnomaly'], 'isNegativeAnomaly': result['isNegativeAnomaly'],
          'isPositiveAnomaly': result['isPositiveAnomaly'], 'upperMargins': result['upperMargins'], 'lowerMargins': result['lowerMargins'],
          'timestamp': [parser.parse(x['timestamp']) for x in sample_data['series']], 
          'value': [x['value'] for x in sample_data['series']]}
    response = pd.DataFrame(data=columns)
    values = response['value']
    label = response['timestamp']
    anomalies = []
    anomaly_labels = []
    index = 0
    anomaly_indexes = []
    p = figure(x_axis_type='datetime', title="Batch Anomaly Detection ({0} Sensitvity)".format(sensitivity), width=800, height=600)
    for anom in response['isAnomaly']:
        if anom == True and (values[index] > response.iloc[index]['expectedValues'] + response.iloc[index]['upperMargins'] or 
                         values[index] < response.iloc[index]['expectedValues'] - response.iloc[index]['lowerMargins']):
            anomalies.append(values[index])
            anomaly_labels.append(label[index])
            anomaly_indexes.append(index)
        index = index+1
    upperband = response['expectedValues'] + response['upperMargins']
    lowerband = response['expectedValues'] -response['lowerMargins']
    band_x = np.append(label, label[::-1])
    band_y = np.append(lowerband, upperband[::-1])
    boundary = p.patch(band_x, band_y, color=Blues4[2], fill_alpha=0.5, line_width=1, legend='Boundary')
    p.line(label, values, legend='Value', color="#2222aa", line_width=1)
    p.line(label, response['expectedValues'], legend='ExpectedValue',  line_width=1, line_dash="dotdash", line_color='olivedrab')
    anom_source = ColumnDataSource(dict(x=anomaly_labels, y=anomalies))
    anoms = p.circle('x', 'y', size=5, color='tomato', source=anom_source)
    p.legend.border_line_width = 1
    p.legend.background_fill_alpha  = 0.1
    show(p, notebook_handle=True)

The following cells call the Anomaly Detector API with two different example time series data sets, and different sensitivities for anomaly detection. Changing the sensitivity of the API to fit the result better for your own scenarios.

Example 1: time series with an hourly sampling frequency

# Hourly Sample
sample_data = json.load(open('sample_hourly.json'))
sample_data['granularity'] = 'hourly'
sample_data['period'] = 24
# 95 sensitivity
build_figure(sample_data,95)


# 90 sensitivity
build_figure(sample_data,90)

#85 sensitivity
build_figure(sample_data,85)

After above cells are executed, you would see following charts showing how sensitivity setting impacts the expected value bounds of example #1.batch_e1_95.pngbatch_e1_90.pngbatch_e1_85.png  

 

Example 2: time series with a daily sampling frequency

#daily sample
sample_data = json.load(open('sample.json'))
sample_data['granularity'] = 'daily'
# 95 sensitivity
build_figure(sample_data,95)

# 90 sensitivity
build_figure(sample_data,90)

# 85 sensitivity
build_figure(sample_data,80)

After above cells are executed, you would see following charts showing how sensitivity setting impacts the expected value bounds of example #2.batch_e2_95.pngbatch_e2_90.pngbatch_e2_85.png  

 

After the code walkthrough, you should have a good understanding how to use the API. For more information, please go to the documentation  site. Especially read the best practices  article first. Feel free to reply to this post if you have further questions!

 

 

1 Comment
Microsoft

Thank you for this blog post @Tony_Xing, I will share this with the people who attended our workshop on deploying custom sci-kit learn anomaly detection solutions to ACI/AKS (https://azure.github.io/LearnAI_Azure_ML/).