Advanced Time Series Anomaly Detector in Fabric
Anomaly Detector, one of Azure AI services, enables you to monitor and detect anomalies in your time series data. This service is being retired by October 2026, and as part of the migration process the anomaly detection algorithms were open sourced and published by a new Python package and we offer a time series anomaly detection workflow in Microsoft Fabric data platform.AI-900: Microsoft Azure AI Fundamentals Study Guide
This comprehensive study guide provides a thorough overview of the topics covered in the Microsoft Azure AI Fundamentals (AI-900) exam, including Artificial Intelligence workloads, fundamental principles of machine learning, computer vision and natural language processing workloads. Learn about the exam's intended audience, how to earn the certification, and the skills measured as of April 2022. Discover the important considerations for responsible AI, the capabilities of Azure Machine Learning Studio and more. Get ready to demonstrate your knowledge of AI and ML concepts and related Microsoft Azure services with this helpful study guide.Multivariate Anomaly Detection in Azure Data Explorer
ADX contains native support for detecting anomalies over multiple time series by using the function series_decompose_anomalies() that can analyze thousands of time series in seconds, enabling near real time monitoring solutions and workflows based on ADX. This function analyzes each metric independently for anomalies, however there are some anomalies that can only be detected by looking on multiple metrics at the same time. In this blog we present new ADX functions for multivariate anomaly detection, that jointly analyze time series of multiple metrics, and present example of these anomalies when analyzing prices of MSFT and SPY pair.
Overview of SR-CNN algorithm in Azure Anomaly Detector
Author: Tony Xing (@XingGuodong), AI Platform, C + AI In the last blog “Introducing Azure Anomaly Detector API”, I didn't provide enough details on one of the algorithms. As the algorithm paper was in the publishing process. The paper was accepted by KDD 2019 for oral presentation later, and this blog serves as an overview of the SR-CNN algorithm and for more details user can always read the paper. By the way, we have a 2-minute video here. Problem definition Before we go into details, let us revisit the problem definition of time series anomaly detection. Challenges For any time-series anomaly detection system that is operating in production with a large scale, there are quite a few challenges, especially on the three areas below: 1. Lack of labels - As you can imagine, with signals generated from clients, services, and sensors every second, the huge amount of volume makes it infeasible to manually label the data. 2. Generalization - With real-world data, there are so many different types of time series with different characteristics, which make it hard to generalize and find a silver bullet to solve all the problems. Some examples can be found in the figure below. 3. Efficiency - For any online anomaly detection system, efficiency is one of the key challenges. The system is expected to have low compute cost and low latency for serving. Inspiration In the computer vision domain, there is this concept called “visual saliency detection”. Saliency is what "stands out" in a photo or scene, enabling our eye-brain to quickly focus on the most important regions, as shown in figures below. Fig. Original image Fig. The salient part of the original image When we look at the time series chart, the most dominant and stand-out part is the anomalies. This similarity is where we got the inspiration and it turned out to generate great results. Algorithm Our solution then borrowed Spectral Residual (SR) from the visual saliency detection domain, then apply CNN on the results produced by the SR model As you can see from the algorithm architecture, after SR transformation, the transformed result magnifies the anomalies and the resulting signal is easier to generalize, therefore it provides us a way to training CNN with synthetic data. Spectral Residual The spectral residual algorithm consists of three major steps: Fourier Transform to get the log amplitude spectrum Calculation of spectral residual Inverse Fourier Transform that transforms the sequence back to the spatial domain Benefits SR is unsupervised, efficient, and has good generality. The problem becomes much easier based on the output of the SR model. We can train CNN on the SR output using fully synthetic data with simple synthetic rule Randomly select several points in the saliency map and calculate the injection value to replace the original point. Result We have performed online and offline experimentation, it outperformed state-of-the-arts consistently on open datasets and internal production datasets.
