End-to-End IoT analytics and machine learning with Azure Data and AI services

By
Published 05-20-2021 07:39 AM 1,379 Views
Microsoft

The Internet of Things (IoT) technology stack in Operational Technology (OT) is widely used across various industries, including oil & gas, manufacturing, utilities, and natural resources, for solving operational challenges and delivering mission-critical insights and analytics.

 

More and more organizations are leveraging Microsoft's Azure cloud platform to perform large scale analytics and machine learning using data from IoT assets, something that had not been simple to do using traditional systems such as Scada and Historians.

 

In this article, we preview an end-to-end Azure Data and AI cloud architecture that enables IoT analytics. This article is based on our 3-part blog series on the Databricks Blog site. You can find more information and code samples starting with

 

Here is the overall architecture discussed in this article and the Databricks blog series:

 

teaserpicture.PNG

For making this article applicable on a common IoT use case, consider the scenario of balancing optimal short-term utilization of an asset, such as a wind turbine, against its long-term maintenance costs.

balance.PNG

In order to develop insights on short-term optimization as well as longer term maintenance costs, various data sources need to be considered and ingested into the cloud for centralized storage and analysis. Here are a few Azure cloud services to consider, depending on whether the data sources can be stream or batch processed.

 

For the wind turbine scenario, streaming data can be the sensor data collected from the turbines, while structured data can be maintenance and failure data collected in a batch process.

 

Once the data sources are ingested into Azure, there are a few options, again depending on stream or batch processing, to process and store the data. In this architecture, the Delta format in Azure Databricks backed by Azure Data Lake Gen 2 is the preferred data format for large-scale IoT data sources: Delta Lake and Delta Engine guide - Azure Databricks - Workspace | Microsoft Docs

 

Once the data is ingested, processed and stored into Delta format, Azure Databricks can be used for big data analytics including data engineering and data science using Spark. As a common pattern, multiple zones within the data lake and aggregations are highlighted below identified through

 

  • Bronze for raw granular IoT data
  • Silver for aggregated data, commonly used for machine learning and data science
  • Gold for enriched data ready for analytics and reporting purposes

Data engineers can use Azure Databricks and create 3 Delta tables corresponding to these three zones. Users can use programming languages namely Python, Scala, R, and SQL in Azure Databricks for accelerated data engineering and data science development.

 

Azure Machine Learning can be used for machine learning, most commonly together with Azure Databricks, in this IoT architecture. For example, Azure Databricks can be used with Spark to engineer features and aggregate data. Then Azure Machine Learning can be used to build models through code, drag-and-drop, or even automated machine learning. In addition, Azure Machine Learning can be used to deploy and operationalize machine learning models.

 

For the wind turbine scenario, the bronze Delta table could be the granular IoT sensor data from the turbines while the silver Delta table is the aggregated (by the hour for example) data. Then Azure Databricks can be used to perform feature engineering and feature selection to build a machine learning and analytics ready dataset. This dataset would then be loaded in Azure Machine Learning to build a predictive maintenance model or a power generation prediction model.

 

Finally, once the predictions and gold enriched data is created in the gold Delta table with Azure Databricks, it can be loaded into Azure Synapse Analytics for BI analytics and reporting scenarios together with Power BI. Azure Data Explorer provides real-time operational analytics so IoT data can be streamed directly from IoT Hub or Event Hub to Data Explorer.

 

In summary, this article covered the end-to-end steps for enabling IoT data analytics and machine learning on the Azure cloud platform, including some best practices, recommended services, and application with wind turbine operations use case. This blog series has the full details and provides code samples as well: Articles by Hubert Duan - The Databricks Blog.

 

 

%3CLINGO-SUB%20id%3D%22lingo-sub-2365938%22%20slang%3D%22en-US%22%3EEnd-to-End%20IoT%20analytics%20and%20machine%20learning%20with%20Azure%20Data%20and%20AI%20services%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2365938%22%20slang%3D%22en-US%22%3E%3CP%3EThe%20Internet%20of%20Things%20(IoT)%20technology%20stack%20in%20Operational%20Technology%20(OT)%20is%20widely%20used%20across%20various%20industries%2C%20including%20oil%20%26amp%3B%20gas%2C%20manufacturing%2C%20utilities%2C%20and%20natural%20resources%2C%20for%20solving%20operational%20challenges%20and%20delivering%20mission-critical%20insights%20and%20analytics.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EMore%20and%20more%20organizations%20are%20leveraging%20Microsoft's%20Azure%20cloud%20platform%20to%20perform%20large%20scale%20analytics%20and%20machine%20learning%20using%20data%20from%20IoT%20assets%2C%20something%20that%20had%20not%20been%20simple%20to%20do%20using%20traditional%20systems%20such%20as%20Scada%20and%20Historians.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20this%20article%2C%20we%20preview%20an%20end-to-end%20Azure%20Data%20and%20AI%20cloud%20architecture%20that%20enables%20IoT%20analytics.%20This%20article%20is%20based%20on%20our%203-part%20blog%20series%20on%20the%20Databricks%20Blog%20site.%20You%26nbsp%3Bcan%20find%20more%20information%20and%20code%20samples%20starting%20with%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EPart%201%3A%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdatabricks.com%2Fblog%2F2020%2F08%2F03%2Fmodern-industrial-iot-analytics-on-azure-part-1.html%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EHow%20to%20Use%20Databricks%20to%20Scale%20Modern%20Industrial%20IoT%20Analytics%20-%20Part%201%20-%20The%20Databricks%20Blog%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3EFull%203-part%20series%20index%3A%26nbsp%3B%3CA%20style%3D%22font-family%3A%20inherit%3B%20background-color%3A%20%23ffffff%3B%22%20href%3D%22https%3A%2F%2Fdatabricks.com%2Fblog%2Fauthor%2Fhubert-dua%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EArticles%20by%20Hubert%20Duan%20-%20The%20Databricks%20Blog%3C%2FA%3E%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EHere%20is%20the%20overall%20architecture%20discussed%20in%20this%20article%20and%20the%20Databricks%20blog%20series%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22teaserpicture.PNG%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F281614i26E40D1ADEE5C60C%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22teaserpicture.PNG%22%20alt%3D%22teaserpicture.PNG%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EFor%20making%20this%20article%20applicable%20on%20a%20common%20IoT%20use%20case%2C%20consider%20the%20scenario%20of%20balancing%20optimal%20short-term%20utilization%20of%20an%20asset%2C%20such%20as%20a%20wind%20turbine%2C%20against%20its%20long-term%20maintenance%20costs.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22balance.PNG%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F281616i0613988660906A56%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22balance.PNG%22%20alt%3D%22balance.PNG%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EIn%20order%20to%20develop%20insights%20on%20short-term%20optimization%20as%20well%20as%20longer%20term%20maintenance%20costs%2C%20various%20data%20sources%20need%20to%20be%20considered%20and%20ingested%20into%20the%20cloud%20for%20centralized%20storage%20and%20analysis.%20Here%20are%20a%20few%20Azure%20cloud%20services%20to%20consider%2C%20depending%20on%20whether%20the%20data%20sources%20can%20be%20stream%20or%20batch%20processed.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EFor%20streaming%20data%20into%20Azure%2C%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fiot-hub%2Fabout-iot-hub%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3EIoT%20Hub%3C%2FA%3E%26nbsp%3Bor%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fevent-hubs%2Fevent-hubs-about%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3EEvent%20Hub%3C%2FA%3E%26nbsp%3Bshould%20be%20used%3CUL%3E%0A%3CLI%3EHere%20is%20a%20reference%20on%20comparing%20these%20two%20services%3A%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fiot-hub%2Fiot-hub-compare-event-hubs%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3ECompare%20Azure%20IoT%20Hub%20to%20Azure%20Event%20Hubs%20%7C%20Microsoft%20Docs%3C%2FA%3E%26nbsp%3B%3C%2FLI%3E%0A%3CLI%3EIn%20general%2C%20for%20connecting%20IoT%20devices%20to%20Azure%2C%20Azure%20IoT%20Hub%20is%20recommended%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3C%2FLI%3E%0A%3CLI%3EFor%20batch%20data%20ingestion%20and%20data%20orchestration%2C%20Azure%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fintroduction%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3EData%20Factory%3C%2FA%3E%20should%20be%20considered%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EFor%20the%20wind%20turbine%20scenario%2C%20streaming%20data%20can%20be%20the%20sensor%20data%20collected%20from%20the%20turbines%2C%20while%20structured%20data%20can%20be%20maintenance%20and%20failure%20data%20collected%20in%20a%20batch%20process.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOnce%20the%20data%20sources%20are%20ingested%20into%20Azure%2C%20there%20are%20a%20few%20options%2C%20again%20depending%20on%20stream%20or%20batch%20processing%2C%20to%20process%20and%20store%20the%20data.%20In%20this%20architecture%2C%20the%20Delta%20format%20in%20Azure%20Databricks%20backed%20by%20Azure%20Data%20Lake%20Gen%202%20is%20the%20preferred%20data%20format%20for%20large-scale%20IoT%20data%20sources%3A%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdatabricks%2Fdelta%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EDelta%20Lake%20and%20Delta%20Engine%20guide%20-%20Azure%20Databricks%20-%20Workspace%20%7C%20Microsoft%20Docs%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EFor%20streaming%20data%20that%20is%20ingested%20into%20IoT%20Hub%20or%20Event%20Hub%2C%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdatabricks%2Fscenarios%2Fwhat-is-azure-databricks%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3EAzure%20Databricks%3C%2FA%3E%20can%20leverage%20Spark%20Streaming%3A%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdatabricks%2Fspark%2Flatest%2Fstructured-streaming%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EStructured%20Streaming%20-%20Azure%20Databricks%20-%20Workspace%20%7C%20Microsoft%20Docs%3C%2FA%3E%26nbsp%3Bto%20consume%20data%20from%20the%20ingestion%20service%20in%20order%20to%20process%20and%20perform%20analysis%26nbsp%3B%3CUL%3E%0A%3CLI%3EAs%20a%20standard%20practice%2C%20Spark%20streaming%20can%20be%20used%20to%20stream%20from%20IoT%20Hub%20or%20Event%20Hub%20to%20a%20Databricks%20Delta%20table%20in%20the%20data%20lake%3A%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdatabricks%2Fdelta%2Fdelta-streaming%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3ETable%20streaming%20reads%20and%20writes%20-%20Azure%20Databricks%20-%20Workspace%20%7C%20Microsoft%20Docs%3C%2FA%3E%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3C%2FLI%3E%0A%3CLI%3EAlternatively%2C%20Data%20Factory%20can%20be%20used%20to%20move%20batch%20data%20to%20the%20data%20lake%20as%20well%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EOnce%20the%20data%20is%20ingested%2C%20processed%20and%20stored%20into%20Delta%20format%2C%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdatabricks%2Fscenarios%2Fwhat-is-azure-databricks%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3EAzure%20Databricks%3C%2FA%3E%26nbsp%3Bcan%20be%20used%20for%20big%20data%20analytics%20including%20data%20engineering%20and%20data%20science%20using%20Spark.%20As%20a%20common%20pattern%2C%20multiple%20zones%20within%20the%20data%20lake%20and%20aggregations%20are%20highlighted%20below%20identified%20through%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EBronze%20for%20raw%20granular%20IoT%20data%3C%2FLI%3E%0A%3CLI%3ESilver%20for%20aggregated%20data%2C%20commonly%20used%20for%20machine%20learning%20and%20data%20science%3C%2FLI%3E%0A%3CLI%3EGold%20for%20enriched%20data%20ready%20for%20analytics%20and%20reporting%20purposes%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EData%20engineers%20can%20use%20Azure%20Databricks%20and%20create%203%20Delta%20tables%20corresponding%20to%20these%20three%20zones.%20Users%20can%20use%20programming%20languages%20namely%20Python%2C%20Scala%2C%20R%2C%20and%20SQL%20in%20Azure%20Databricks%20for%20accelerated%20data%20engineering%20and%20data%20science%20development.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAzure%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fmachine-learning%2Foverview-what-is-azure-ml%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3EMachine%20Learning%3C%2FA%3E%26nbsp%3Bcan%20be%20used%20for%20machine%20learning%2C%20most%20commonly%20together%20with%20Azure%20Databricks%2C%20in%20this%20IoT%20architecture.%20For%20example%2C%20Azure%20Databricks%20can%20be%20used%20with%20Spark%20to%20engineer%20features%20and%20aggregate%20data.%20Then%20Azure%20Machine%20Learning%20can%20be%20used%20to%20build%20models%20through%20code%2C%20drag-and-drop%2C%20or%20even%20automated%20machine%20learning.%20In%20addition%2C%20Azure%20Machine%20Learning%20can%20be%20used%20to%20deploy%20and%20operationalize%20machine%20learning%20models.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFor%20the%20wind%20turbine%20scenario%2C%20the%20bronze%20Delta%20table%20could%20be%20the%20granular%20IoT%20sensor%20data%20from%20the%20turbines%20while%20the%20silver%20Delta%20table%20is%20the%20aggregated%20(by%20the%20hour%20for%20example)%20data.%20Then%20Azure%20Databricks%20can%20be%20used%20to%20perform%20feature%20engineering%20and%20feature%20selection%20to%20build%20a%20machine%20learning%20and%20analytics%20ready%20dataset.%20This%20dataset%20would%20then%20be%20loaded%20in%20Azure%20Machine%20Learning%20to%20build%20a%20predictive%20maintenance%20model%20or%20a%20power%20generation%20prediction%20model.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFinally%2C%20once%20the%20predictions%20and%20gold%20enriched%20data%20is%20created%20in%20the%20gold%20Delta%20table%20with%20Azure%20Databricks%2C%20it%20can%20be%20loaded%20into%20Azure%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fsynapse-analytics%2Foverview-what-is%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3ESynapse%20Analytics%3C%2FA%3E%26nbsp%3Bfor%20BI%20analytics%20and%20reporting%20scenarios%20together%20with%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fpower-bi%2Ffundamentals%2Fpower-bi-overview%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3EPower%20BI%3C%2FA%3E.%20Azure%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-explorer%2Fdata-explorer-overview%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3EData%20Explorer%3C%2FA%3E%20provides%20real-time%20operational%20analytics%20so%20IoT%20data%20can%20be%20streamed%20directly%20from%20IoT%20Hub%20or%20Event%20Hub%20to%20Data%20Explorer.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20summary%2C%20this%20article%20covered%20the%20end-to-end%20steps%20for%20enabling%20IoT%20data%20analytics%20and%20machine%20learning%20on%20the%20Azure%20cloud%20platform%2C%20including%20some%20best%20practices%2C%20recommended%20services%2C%20and%20application%20with%20wind%20turbine%20operations%20use%20case.%20This%20blog%20series%20has%20the%20full%20details%20and%20provides%20code%20samples%20as%20well%3A%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdatabricks.com%2Fblog%2Fauthor%2Fhubert-dua%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EArticles%20by%20Hubert%20Duan%20-%20The%20Databricks%20Blog%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-2365938%22%20slang%3D%22en-US%22%3E%3CP%3EInterested%20in%20doing%20IoT%20analytics%20and%20machine%20learning%20using%20the%20Azure%20platform%3F%20Come%20and%20explore%20with%20us%20the%20various%20Azure%20Data%20and%20AI%20services%20to%20enable%20this%20popular%20scenario.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22teaserpicture.PNG%22%20style%3D%22width%3A%20593px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F281836iDEAB59688103AF11%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22teaserpicture.PNG%22%20alt%3D%22teaserpicture.PNG%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2365938%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAdvance%20Analytics%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EArtificial%20Intelligence%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EData%20Platform%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Co-Authors
Version history
Last update:
‎May 19 2021 07:44 AM
Updated by: