Detecting Methane Leaks using Azure Metrics Advisor

At Microsoft, we are committed to empowering our customers and partners in their journey to achieve net-zero emissions. As part of our commitment to sustainability, both for our own operations and across our value chain. We are also working with customers in many industries to help them achieve their sustainability goals. As part of that effort, in this blog post, we show how Azure Metrics Advisor can be used to detect anomalous methane emissions in the oil and gas industry.

Methane (CH₄) is a potent greenhouse gas that is much more effective at absorbing infrared radiation than CO₂, and therefore, has a severe global warming impact. The Global Warming Potential (GWP) of a greenhouse gas is its ability to trap extra heat in the atmosphere over time relative to CO₂. The Intergovernmental Panel on Climate Change (IPCC) estimates the 20-year GWP of methane to be more than 80 times that of CO₂, indicating its potency as a greenhouse gas. This underscores the importance of detection and remediation of methane leaks into the atmosphere.  

A primary source of methane leaks into the atmosphere is the global oil and gas industry. From production and processing to transmission, storage, and distribution. The International Energy Agency (IEA) estimates that it is technically possible to avoid around 75% of today’s methane emissions from global oil and gas operations. Furthermore, a Stanford study estimated that the largest 5% of leaks (“super emitters”) typically contribute over 50% of the total leakage volume. Detecting these leaks, and remediating them, can have a significant environmental impact. Reducing natural gas emissions by 30% just within the Permian Basin (one of the most prolific Shale Basins in the United States) is equivalent to removing 5 million cars from the roads over 20 years. Additionally, a tightening regulatory environment and pressure from increasingly influential Environmental, Social, and Governance (ESG) investment trends are driving a renewed effort to detect anomalous methane emissions more accurately.

IoT sensor grids are one of the main technologies that the Energy industry is leveraging to detect methane leaks close to oil and gas facilities in real-time. Sensors deployed near oil and gas facilities can detect both intended (routine) emissions and unintended (fugitive) emissions that commonly occur from equipment such as pneumatic devices, storage tanks, and compressors. Additionally, methane concentrations at a particular location exhibit seasonality and are affected by temperature, relative humidity, wind direction, and wind speed among other factors. Therefore, other ground sensors that measure these environmental variables might be required.

By ingesting IoT sensor data to the cloud in real-time, it is possible to enable custom analytics that can detect sudden anomalies in the levels of methane measured by these ground sensors. This process needs to simultaneously satisfy multiple technical requirements around latency, concurrency, and scalability and Azure provides an excellent platform to help achieve this. Azure IoT Hub enables highly secure and reliable ingestion of data from IoT sensors deployed in the field. Additionally, Azure Metrics Advisor provides a powerful time-series monitoring platform that offers a set of APIs and a web-based workspace for ingesting time-series data, performing univariate and multivariate anomaly detection, sending alerts through multiple channels, and diagnosing the root cause of anomalous incidents. Azure Metrics Advisor works great in streaming scenarios, such as methane leak detection, where detecting anomalies in real-time is critical.  It allows the detection of spikes, dips, deviations from cyclic patterns, and trend changes through both univariate and multivariate anomaly detection APIs.

In the remainder of this post, we will show how you can ingest your IoT sensor data into Azure, and onboard your data to Azure Metrics Advisor to detect anomalies.

Data ingestion and processing

The first step is to deploy methane sensors oil and gas facilities such as pipelines, storage tanks, and compressors, in addition to sensors that measure other atmospheric variables such as wind speed, wind direction, temperature, and relative humidity. These sensors can be placed optimally, using a machine learning framework developed by Microsoft, that aims to maximize detection of possible methane leaks with a limited sensor budget. Once the sensors are deployed, Azure IoT Hub can be used to regularly communicate with the sensors and ingest the data into a data store on Azure such as Azure SQL database, Azure Data Explorer (ADX), or Azure CosmosDB. An example architecture diagram is shown below.

A sample architecture diagram for the solution

Data onboarding to Azure Metrics Advisor

Once the data is flowing to the data store, a data feed can be set up in   to onboard the data into Metrics Advisor and unlock many of the powerful capabilities the Azure Metrics Advisor provides. Azure Metrics Advisor can authenticate and pull data from a variety of data sources such as Azure Blob Storage, Azure CosmosDB, Azure SQL database, and PostgreSQL. For detailed documentation on this data onboarding process, please refer to this Azure Metrics Advisor documentation page.

Metrics Advisor supports a wide variety of data sources

Anomaly detection in Azure Metrics Advisor

After a data feed is set up in Azure Metrics Advisor, it is easy to add an anomaly detection configuration with custom rules that best fit this scenario. For example, a user might only be interested in anomalous spikes in the methane measurements (that might correspond to leaks) especially if the wind speed is above a certain range. Similarly, the user might not be interested if the methane measurements suddenly drop. It is very easy to set up such custom rules in Metrics Advisor to ensure that the correct anomalies are detected. The following image shows the detected anomalies in a synthetic methane sensing scenario highlighted in red.

Once a detection configuration is set up, detected anomalies are highlighted in red.

Incident hub in Metrics Advisor allows the user to see an overview of all the latest detected anomalies in real-time and see their severity scores. It also allows the user to diagnose and analyze the root cause of each incident.

Incident hub allows the user to see and diagnose all the detected anomalies in one place

Furthermore, the incident hub provides a powerful time-series clustering feature that allows the user to cluster signals from different sensors together to better understand and diagnose anomalous incidents. In the case of methane leak detection, this can help the operator understand which sensors have detected a leak, and hone in on the likely source of that leak.

Time-series clustering in Metrics Advisor can help the user diagnose the root cause of a methane leak

In the case of multivariate anomaly detection (MVAD), the multivariate anomaly detector API can detect anomalies in the interactions among up to 300 variables. In this scenario, these variables can correspond to methane concentration measurements from various sensors, in addition to wind speed and wind direction, ambient temperature, relative humidity, and any other variable of interest. For more on the difference between univariate and multivariate anomaly detection, please visit this page. In the case of methane leak detection, a spike in methane concentration may or may not be an indication of an anomalous emission based partly on other variables such as wind speed. When the target variable is hidden (in this case the target variable is the rate that methane is leaking from a source, also known as methane flux rate) it is often useful to use the MVAD API to detect anomalies in variables that cannot be directly measured. The MVAD API can be used directly, or through an upcoming web interface that requires no coding experience

Setting up alerts and taking action

When anomalies are detected, it is very useful to set up automated alerts that notify the operator of a possible methane leak, and prompt them to investigate and remediate the leak. Metrics Advisor allows for easily setting up multiple alerting channels such as email, Microsoft Teams, or even webhooks. Metrics advisor also allows the user to configure the alert based on the severity of the detected anomalies. For example, the user can choose to only receive alerts of high severity or choose a different alerting mechanism for anomalies of low severity that might not require immediate attention compared to high severity anomalies.

Metrics Advisor supports many alerting channels including webhooks

Azure Metrics Advisor can be used through either its web-based workspace (like we demonstrated in this blog post) or through its software development kit (SDK). For more details, please refer to this documentation page. If you are currently using an Apache Spark environment, you can access some of the univariate and multivariate anomaly detection capabilities provided by Azure Anomaly Detector through the open source SynapseML library. Please refer to this post for more details.

Get started today

To get started, go to the Azure portal and create a new Azure Metrics Advisor resource. You can also read the Metrics Advisor documentation to learn more about the capabilities of this service.

For more information on Azure Anomaly Detector or Azure Metrics Advisor, please contact:  AnomalyDetector@microsoft.com or MetricsAdvisor@microsoft.com. For more information on our work around methane leak detection, please contact babak.biglari@microsoft.com.