Anomaly Detector was on public preview on 3/26. We are pleased to see the adoption from a variety of customers across different industry verticals. We’d like to use this blog to provide additional details on Anomaly Detector and how to use it to achieve the best result with code walkthrough.
Overview
Anomaly detection aims to discover unexpected events or rare items in data. Accurate anomaly detection leads to prompt troubleshooting, which helps to avoid revenue loss and maintain brand reputation. At Microsoft, hundreds of teams rely on the technology we have built to monitor millions of metrics from Bing, Office, and Azure.
Anomaly Detector provides two APIs that detect anomalies automatically in time series with simple parameters, which require no machine learning background. It is designed for the scenarios of operational monitoring, business KPI monitoring, and IoT monitoring. Using Anomaly Detector APIs, you can infuse anomaly detection capabilities into your own platform and business process.
Who can use Anomaly Detector
Anomaly Detector is designed for anyone who can make a REST API call with script or code. It hides details of anomaly detection algorithms into only one main parameter to adjust the sensitivity of the detection. That makes it a perfect fit for integrating Microsoft anomaly detection capabilities into your business applications if you are tired of tuning the parameters for your AI algorithms or if you have no machine learning background.
Highlights
A gallery of algorithms
State of art anomaly detection system often uses a one size fit all approach. Which mean they apply some specific algorithm on all types of time series. Our learning is that each algorithm can handle some specific type of time series better. Our innovation is that we provide a generic framework to plug in different algorithm ensembles to handle a wide spectrum of different time series. In our different ensembles, we have following algorithms used.
- Fourier Transformation
- Extreme Studentized Deviate (ESD)
- STL Decomposition
- Dynamic Threshold
- Z-score detector
- SR-CNN
The following picture shows the algorithm selecting flow of Anomaly Detector. We will use another blog for more details on the algorithms.
Detecting all kinds of anomalies through one single endpoint
Besides spikes and dips, Anomaly Detector also detects many other kinds of anomalies, such as trend change and off-cycle softness, all in one single API endpoint.
Maintain simplicity outside: One parameter tuning
While running the Anomaly Detection service in Microsoft for several years, we have kept improving the one parameter strategy to make the tuning easier. The main parameter you need to customize is “Sensitivity”, which is from 1 to 99 to adjust the outcome to fit the scenario. Other advanced parameters are reserved for data scientists for further tuning.
Maintain sophistication inside: Selecting / Detecting / Filtering
To support one parameter tuning, we have put together a gallery of anomaly detection algorithms and designed a sophisticated strategy to leverage the collective power of machine learning to gain good precision and recall.
When we get the time series from an API request, the first thing is to extract features from the time series, for example continuity, mean, STD, trend, and period. With those features, the system selects the most appropriate algorithm.
It is not easy to find a good balance in recall and precision in one single detection without enough context. Anomaly Detector separates improving recall and precision into two steps. Firstly, it makes sure the recall is high. Then, it uses those features, which are extracted from the time series plus the sensitivity, which is the only parameter provided by the caller, to apply filtering on the results. This strategy has proved to provide better results.
Code walkthrough
The best way to show you the code and results is through Jupyter notebook, which you can see the code and visualization side by side. Let's go.
Simulate a streaming detection with "last" api
You can find the related sample data and Jupyter notebook here: https://github.com/Azure-Samples/anomalydetector/tree/master/ipython-notebook . Make sure you do read the readme here first.
Let's start with Latest point detection with Anomaly Detector API.ipynb notebook and walk through the code cell by cell.
By calling the API on your data's latest points, you can monitor your data as it's created.
The following example simulates using the Anomaly Detector API on streaming data. Sections of the example time series are sent to the API over multiple iterations, and the anomaly status of each section's last data point is saved. The data set used in this example has a pattern that repeats roughly every seven data points (the period in the request's JSON file), so for best results, the data set is sent in groups of 29 points (4 * period + an extra data point.).
Fill in the API access key and endpoint you received after creating your Anomaly Detector resource in the code block below. Refer here on how to create AD resource.
The following request URLs must be combined with the appropriate endpoint for your subscription. For example:
https://westus2.api.cognitive.microsoft.com/anomalydetector/v1.0/timeseries/entire/detect
Batch detection
To detect anomalies throughout a batch of data points over a given time range, use the following request URI with your time series data:
/timeseries/entire/detect
.
By sending your time series data at once, the API will generate a model using the entire series, and analyze each data point with it.
Streaming detection
To continuously detect anomalies on streaming data, use the following request URI with your latest data point:
/timeseries/last/detect'
.
By sending new data points as you generate them, you can monitor your data in real time. A model will be generated with the data points you send, and the API will determine if the latest point in the time series is an anomaly.
# To start sending requests to the Anomaly Detector API, paste your subscription key below,
# and replace the endpoint variable with the endpoint for your region.
subscription_key = ''
endpoint_latest = 'https://westus2.api.cognitive.microsoft.com/anomalydetector/v1.0/timeseries/last/detect'
You import necessary libraries for numerical calculation and visualization in the following code block.
import requests
import json
import pandas as pd
import numpy as np
from __future__ import print_function
import warnings
warnings.filterwarnings('ignore')
# Import library to display results
import matplotlib.pyplot as plt
%matplotlib inline
from bokeh.plotting import figure,output_notebook, show
from bokeh.palettes import Blues4
from bokeh.models import ColumnDataSource,Slider
import datetime
from bokeh.io import push_notebook
from dateutil import parser
from ipywidgets import interact, widgets, fixed
output_notebook()
The code block below shows you a detect function, which takes three parameters as inputs and returns the api call response in json data.
The three parameters are
- The api endpoint you defined earlier
- subscription_key you defined earlier
- request_data the time series data you post to the api point
The return data is the meta data related to the anomaly detection results that you can find in the api definition.
def detect(endpoint, subscription_key, request_data):
headers = {'Content-Type': 'application/json', 'Ocp-Apim-Subscription-Key': subscription_key}
response = requests.post(endpoint, data=json.dumps(request_data), headers=headers)
if response.status_code == 200:
return json.loads(response.content.decode("utf-8"))
else:
print(response.status_code)
raise Exception(response.text)
The code block below shows you a build_figure function, which takes three parameters as inputs and do the visualization upon extracted results, that is, sample data, anomalies, upper bound and lower bound, expected values...
def build_figure(result, sample_data, sensitivity):
columns = {'expectedValues': result['expectedValues'], 'isAnomaly': result['isAnomaly'], 'isNegativeAnomaly': result['isNegativeAnomaly'],
'isPositiveAnomaly': result['isPositiveAnomaly'], 'upperMargins': result['upperMargins'], 'lowerMargins': result['lowerMargins']
, 'value': [x['value'] for x in sample_data['series']], 'timestamp': [parser.parse(x['timestamp']) for x in sample_data['series']]}
response = pd.DataFrame(data=columns)
values = response['value']
label = response['timestamp']
anomalies = []
anomaly_labels = []
index = 0
anomaly_indexes = []
p = figure(x_axis_type='datetime', title="Anomaly Detection Result ({0} Sensitivity)".format(sensitivity), width=800, height=600)
for anom in response['isAnomaly']:
if anom == True and (values[index] > response.iloc[index]['expectedValues'] + response.iloc[index]['upperMargins'] or
values[index] < response.iloc[index]['expectedValues'] - response.iloc[index]['lowerMargins']):
anomalies.append(values[index])
anomaly_labels.append(label[index])
anomaly_indexes.append(index)
index = index+1
upperband = response['expectedValues'] + response['upperMargins']
lowerband = response['expectedValues'] -response['lowerMargins']
band_x = np.append(label, label[::-1])
band_y = np.append(lowerband, upperband[::-1])
boundary = p.patch(band_x, band_y, color=Blues4[2], fill_alpha=0.5, line_width=1, legend='Boundary')
p.line(label, values, legend='value', color="#2222aa", line_width=1)
p.line(label, response['expectedValues'], legend='expectedValue', line_width=1, line_dash="dotdash", line_color='olivedrab')
anom_source = ColumnDataSource(dict(x=anomaly_labels, y=anomalies))
anoms = p.circle('x', 'y', size=5, color='tomato', source=anom_source)
p.legend.border_line_width = 1
p.legend.background_fill_alpha = 0.1
show(p, notebook_handle=True)
The code block below shows you a detect_anomaly function, which takes sensitivity as input and repeatly call the detect function to simulate monitor streaming data as new data is created.
def detect_anomaly(sensitivity):
sample_data = json.load(open('sample.json'))
points = sample_data['series']
skip_point = 29
result = {'expectedValues': [None]*len(points), 'upperMargins': [None]*len(points),
'lowerMargins': [None]*len(points), 'isNegativeAnomaly': [False]*len(points),
'isPositiveAnomaly':[False]*len(points), 'isAnomaly': [False]*len(points)}
anom_count = 0
for i in range(skip_point, len(points)+1):
single_sample_data = {}
single_sample_data['series'] = points[i-29:i]
single_sample_data['granularity'] = 'daily'
single_sample_data['maxAnomalyRatio'] = 0.25
single_sample_data['sensitivity'] = sensitivity
single_point = detect(endpoint_latest, subscription_key, single_sample_data)
if single_point['isAnomaly'] == True:
anom_count = anom_count + 1
result['expectedValues'][i-1] = single_point['expectedValue']
result['upperMargins'][i-1] = single_point['upperMargin']
result['lowerMargins'][i-1] = single_point['lowerMargin']
result['isNegativeAnomaly'][i-1] = single_point['isNegativeAnomaly']
result['isPositiveAnomaly'][i-1] = single_point['isPositiveAnomaly']
result['isAnomaly'][i-1] = single_point['isAnomaly']
build_figure(result, sample_data, sensitivity)
The following cells call the Anomaly Detector API with an example time series data set and different sensitivities for anomaly detection. Varying the sensitivity of the Anomaly Detector API can improve how well the response fits your data.
# 95 sensitvity
detect_anomaly(95)
# 85 sensitvity
detect_anomaly(85)
After all cells are executed, you would see following charts showing how sensitivity setting impacts the expected value bounds. The sensitivity is a way we allow users to tune the model with the understanding that different business or person could have different tolerance on even the same dataset. The higher the sensitivity, the narrower the band, as a result, you would get more anomalies.
Detect all anomalies within time series data in a batch with "entire" API
Let's switch to the Batch anomaly detection with Anomaly Detector API.ipynb notebook and walk through the code cell by cell.
This notebook shows you how to send a batch anomaly detection request, and visualize the anomalies found throughout the example data set. The graph created at the end of this notebook will display the following items:
- Anomalies found throughout the data set, highlighted.
- The expected values versus the values contained in the data set.
- Anomaly detection boundaries
You need to fill in the API access key and endpoint you received after creating your Anomaly Detector resource in the code block below.
# To start sending requests to the Anomaly Detector API, paste your subscription key below,
# and replace the endpoint variable with the endpoint for your region.
subscription_key = ''
endpoint = 'https://westus2.api.cognitive.microsoft.com/anomalydetector/v1.0/timeseries/entire/detect'
You import necessary libraries for numerical calculation and visualization in the following code block.
import requests
import json
import pandas as pd
import numpy as np
from __future__ import print_function
import warnings
warnings.filterwarnings('ignore')
# Import library to display results
import matplotlib.pyplot as plt
%matplotlib inline
from bokeh.plotting import figure,output_notebook, show
from bokeh.palettes import Blues4
from bokeh.models import ColumnDataSource,Slider
import datetime
from bokeh.io import push_notebook
from dateutil import parser
from ipywidgets import interact, widgets, fixed
output_notebook()
The code block below shows you a detect function, which takes three parameters as inputs and returns the api call response in json data.
The three parameters are
- The api endpoint you defined earlier
- subscription_key you defined earlier
- request_data the time series data you post to the api point
The return data is the meta data related to the anomaly detection results that you can find in the api definition.
def detect(endpoint, subscription_key, request_data):
headers = {'Content-Type': 'application/json', 'Ocp-Apim-Subscription-Key': subscription_key}
response = requests.post(endpoint, data=json.dumps(request_data), headers=headers)
if response.status_code == 200:
return json.loads(response.content.decode("utf-8"))
else:
print(response.status_code)
raise Exception(response.text)
The code block below shows you a build_figure function, which takes two parameters as inputs and do the visualization upon extracted results, that is, sample data, anomalies, upper bound and lower bound, expected values...
def build_figure(sample_data, sensitivity):
sample_data['sensitivity'] = sensitivity
result = detect(endpoint, subscription_key, sample_data)
columns = {'expectedValues': result['expectedValues'], 'isAnomaly': result['isAnomaly'], 'isNegativeAnomaly': result['isNegativeAnomaly'],
'isPositiveAnomaly': result['isPositiveAnomaly'], 'upperMargins': result['upperMargins'], 'lowerMargins': result['lowerMargins'],
'timestamp': [parser.parse(x['timestamp']) for x in sample_data['series']],
'value': [x['value'] for x in sample_data['series']]}
response = pd.DataFrame(data=columns)
values = response['value']
label = response['timestamp']
anomalies = []
anomaly_labels = []
index = 0
anomaly_indexes = []
p = figure(x_axis_type='datetime', title="Batch Anomaly Detection ({0} Sensitvity)".format(sensitivity), width=800, height=600)
for anom in response['isAnomaly']:
if anom == True and (values[index] > response.iloc[index]['expectedValues'] + response.iloc[index]['upperMargins'] or
values[index] < response.iloc[index]['expectedValues'] - response.iloc[index]['lowerMargins']):
anomalies.append(values[index])
anomaly_labels.append(label[index])
anomaly_indexes.append(index)
index = index+1
upperband = response['expectedValues'] + response['upperMargins']
lowerband = response['expectedValues'] -response['lowerMargins']
band_x = np.append(label, label[::-1])
band_y = np.append(lowerband, upperband[::-1])
boundary = p.patch(band_x, band_y, color=Blues4[2], fill_alpha=0.5, line_width=1, legend='Boundary')
p.line(label, values, legend='Value', color="#2222aa", line_width=1)
p.line(label, response['expectedValues'], legend='ExpectedValue', line_width=1, line_dash="dotdash", line_color='olivedrab')
anom_source = ColumnDataSource(dict(x=anomaly_labels, y=anomalies))
anoms = p.circle('x', 'y', size=5, color='tomato', source=anom_source)
p.legend.border_line_width = 1
p.legend.background_fill_alpha = 0.1
show(p, notebook_handle=True)
The following cells call the Anomaly Detector API with two different example time series data sets, and different sensitivities for anomaly detection. Changing the sensitivity of the API to fit the result better for your own scenarios.
Example 1: time series with an hourly sampling frequency
# Hourly Sample
sample_data = json.load(open('sample_hourly.json'))
sample_data['granularity'] = 'hourly'
sample_data['period'] = 24
# 95 sensitivity
build_figure(sample_data,95)
# 90 sensitivity
build_figure(sample_data,90)
#85 sensitivity
build_figure(sample_data,85)
After above cells are executed, you would see following charts showing how sensitivity setting impacts the expected value bounds of example #1
.
Example 2: time series with a daily sampling frequency
#daily sample
sample_data = json.load(open('sample.json'))
sample_data['granularity'] = 'daily'
# 95 sensitivity
build_figure(sample_data,95)
# 90 sensitivity
build_figure(sample_data,90)
# 85 sensitivity
build_figure(sample_data,80)
After above cells are executed, you would see following charts showing how sensitivity setting impacts the expected value bounds of example #2
.
After the code walkthrough, you should have a good understanding how to use the API. For more information, please go to the documentation site. Especially read the best practices article first. Feel free to reply to this post if you have further questions!