Artificial Intelligence is delivering unprecedented efficiencies in manufacturing. From optimizing operations to streamlining the supply chain to improving how employees work. Techniques like Machine Learning (ML) are a major component in most manufacturing transformation projects today.
Applying Machine Learning to almost any problem is getting easier every day, but it requires a disciplined approach to evaluate a use case and build repeatable processes to scale these techniques beyond a single use case. In this article we will discuss an approach on how to evaluate use cases faster and build this repeatable process. Operationalizing machine learning projects require continuous experimentation and multiple components, for this article we will focus on aspects to accelerate ML lifecycle for Industrial IoT projects.
A typical machine learning journey consists of three high level phases, that are executed in loop for continuous experimentation. Each of these phases answers a certain set of questions that are critical to understanding whether machine learning is the right approach to solve specific business problems and if it is, how to go about building and managing machine learning models. Here are some example questions for each phase:
|
Explore |
Experiment |
Operate |
|
Problem Definition - What is the business case? - What is the business impact? Data Acquisition - Is the data confidential? - Is historical data available? - Is the data relevant? - Is the data enough? - Is there a versioned dataset to start exploration? Exploratory Data Analysis - Is the data normalized? - Is the data distribution normal? - Is there a strong positive or negative correlation between features and target? - Is there a need to engineer new features? - Is the data collected at the right frequency? - Is there a versioned and labeled dataset ready for modeling? |
Business Alignment - Is the model objective aligned with business objective? Model & Data Management - Is the labeled dataset split evenly between train, test, and validation datasets? - Is the baseline modeling done with different algorithm(s)? - Are the model experiments tracked? - Are the model(s) registered? Training Pipeline - Is the model training automated? - Is the model training parameterized for multi-model retraining?
|
Business Actions Mapping - Does the business action require real-time prediction? - Does business action require prediction on the edge? Deployment & Monitoring - Is the model’s performance monitored? - Are the predicted results logged, to compare against actuals? - Is the model(s) deployment automated?
|
Now let’s see some techniques to answer these questions and start building a machine learning solution with an example business case. All the sample code is available in our industrial-iot-patterns GitHub repo, along with many other samples to help implement IIoT and AI scenarios in Azure.
Explore – Problem Definition
Business Case
- The average scrap rate per shift is around (x)% even with measures to improve yield.
- Reducing scrap and understanding what variables are contributing to scrap is a critical business goal.
- Manufacturing lines are connected to PLC(s) that can provide variable data.
- Product quality check takes around 24hrs. and quality results are logged in a different system.
Business Impact
- (x)% reduction in scrap == $(y) in savings
High Level Potential Solution
- Build a model to predict quality failures in minutes.
- Help understand what parameters are contributing to the quality failures.
- Recommend actions to reduce quality failures.
- Recommend additional parameters or data collection methods to look at.
Sample Design:
- Machine Data
- Generated via an IoT Edge Module
- Contains 40+ different sensor values
- Contains production batch number
- Production Quality Data
- Contains production batch number
- Contains quality error code for each batch
- 1 = Meets quality expectations | 0 = Does not meet quality expectations.
To generate the data, you can either follow the given prerequisites or add the SimulatedSensors module in your IoT Edge device. Here is the high-level design:
Explore - Data Acquisition
For IIoT projects we often need to merge the near real-time telemetry data with data from other systems like MES, ERP Quality Management, etc. to build the initial dataset for modeling. Typically, this step may require heavy compute and scalable post processing storage. Hence, we are using Azure ML managed compute to process both the telemetry data directly from Data Explorer and merge that with the Production Quality Data from Data Lake.
Once you have the ML workspace and datastore created, run the 1_create_raw_dataset.ipynb notebook to see the merged dataset in ML Studio. The notebook uses kqlmagic extension to fetch the sensor data directly data explorer.
Explore - Exploratory Data Analysis
Exploratory Data Analysis (EDA) is the first step before we build any custom models using machine learning. This is a critical and often complex step wherein we normalize & clean the data, understand data distribution, outliers, correlations and assess the data for various hypothesis and experiments. For our scenario we will do:
- Data distribution and Correlation Analysis: This helps us understand which sensors are strongly correlated to predict the quality output. Run the 2_exploratory_analysis_feature_selection.ipynb notebook.
Below histograms display the data distribution for each of our sensor values. Ideally, we are looking for sensors closer to a normal distribution.
Below correlation heat map shows the positive or negative correlation among the sensor values. Ideally, we are looking to understand correlations between our sensors and the output that we are trying to predict, Quality.
- Frequency Analysis: This helps us understand optimal frequency value (in Hz) at which we should collect the sensor values. 1Hz equals to 1 sensor value per second, 10Hz equals to 10 sensor values per second and so on. Below line graphs show various frequency comparisons and ideally, we are looking for if there are strong reasons to increase or decrease our data capture rate for each of the sensors. Run the 2_frequency_analysis.ipynb notebook.
- Baseline Modeling: This helps us understand which types of algorithms are best suited for the given dataset. Run the 3_baseline_modeling-automl.ipynb and 3_baseline_modeling.ipynb notebooks.
Automated ML can help identify algorithms that best fit our data, along with providing an interactive dashboard to understand model performance, top features and extract patterns from our data.
Below table shows the list of all the algorithms that we used for our baseline modeling, along with the model evaluation metrics. For our use case we are optimizing for f1-score and recall. See the ML objectives section for more details on how to determine which metric is right for your use case
Experiment
Continuous experimentation is key to building machine learning models with high accuracy of its predictions. This often requires periodic retraining of your model with new data. It is also important to log each of these retraining experiments to compare model improvement over time and track model versions. So, to perform these experiments we will build an Azure Machine Learning Pipeline. Azure Machine Learning pipeline is an independently executable workflow of a complete machine learning task. An Azure Machine Learning pipeline helps to standardize the best practices of producing a machine learning model, enables the team to execute at scale, and improves the model building efficiency. Follow steps from here to build the pipeline.
The pipeline has two steps:
- build-datasets: Builds and registers train and test datasets.
- build-model: Builds and registers a new model based on the features provided as a dynamic parameter.
This endpoint can now be triggered manually or automatically from tools like Synapse Pipelines or Logic Apps.
Since the endpoint uses a parameter to select the sensors for modeling, we can now build multiple sensor models using a single pipeline endpoint.
Each pipeline run job logs all the outputs so it’s easier to track and debug.
Operate
Once the model is trained and registered it’s time to deploy and test our models against live data, also known as inferencing. For our scenario we want quality predictions every few minutes as the production cycle takes some time.
So, we built another pipeline to generate predictions. Follow steps from here
It has only one step but multiple parameters to dynamically fetch live data from data explorer and generate the prediction output.
We can then integrate the model prediction pipeline with Synapse Pipelines to automatically trigger on a schedule and store the output predictions in a Database. Follow steps from here.
We can use the stored prediction outputs to compare against the actuals and see how our model is improving over time or if it needs retraining.
Conclusion
Machine Learning is a major component in most manufacturing transformation projects today. A key skill for a machine learning project is the ability to understand the data coming out of both the IT and OT systems. Most customers do have experts who understand their domains very well, but they lack the skills to apply these new machine learning techniques to better reason about the data. If these domain experts are enabled with the right set of tools and techniques, they can amplify their skills. I hope the above approach helps to sharpen these skills.
To learn more see our Industrial IoT patterns doc and deployment samples. As always, feel free to share your thoughts and ask your questions in the comments!