In manufacturing as well as other industries, safety, like logistics, needs special attention. Not only can a lack of safety mechanisms result in downtime and costly maintenance, but also, even worse, in physical harm to the staff operating the facilities. One of these safety obstacles is caused by loose screws and bolts. Constant vibrations, corrosion, and mistakes made during the assembly are the main drivers for a declining preload force of screws. Considering the harsh environment present in manufacturing and logistics, it’s only a matter of time until the first screw connections will fail.
Therefore, CANCOM developed a solution detecting loose screws and bolts using Azure Percept, while participating in the Azure Percept Bootcamp. This blog post shows how the solution works using the example of a drainage channel.
Solution architecture
Our continuous quality control solution monitors the state of screwed assemblies using Azure Percept. The following is the architecture for the solution.
Core components
The core components of the solution are:
|
Component |
Description |
|
Captures the data stream from the vision module of the Azure Percept Dev Kit and uses a pre-built AI model to detect the state of screwed assemblies. |
|
|
Handles the communication with the IoT Hub in the cloud. |
|
|
Service for training the AI model. |
|
|
Management portal for an easy development and management experience of the Azure Percept solution. |
|
|
Receives the data from the modules running on the edge and enables the management of all workloads on the edge. |
|
|
Processes the raw data from the IoT Hub and stores it in the Azure SQL Database. |
|
|
Storage to persist the data collected. |
|
|
Visualizes the data from the Azure SQL Database. Note that it is also possible to either skip the database or add a second output from the Stream Analytics service directly to Power BI to achieve dashboards consuming stream data and delivering near real-time insights on the current state of the monitored facilities. |
Solution Implementation
For simulation of a channel drainage, a demo setup as shown below was used.
After setting up the Azure Percept Dev Kit, the other components can be deployed and configured.
First, the database is created with a table of the desired telemetry data schema. Then, an IoT Hub with a dedicated consumer group for the Stream Analytics service is needed. After that, the IoT Hub can be configured as a stream input in the Stream Analytics service. The Stream Analytics service uses its query to format the raw data of the IoT Hub and saves it in the database configured as output.
Training the model
As previously mentioned, the training of the AI-model was done using Azure Cognitive Services. We created a custom vision model and gathered our first images using the Image capture functionality in Azure Percept Studio.
We then proceeded to tag the images with these tags:
|
Tag |
Description |
|
bolt ok |
A fully tightened screw or bolt |
|
bolt loose |
A screw or bolt that came loose. Usually where the thread of the screw was visible |
|
bolt |
A screw that is not inside a socket |
|
empty screw hole |
An empty hole missing a screw or bolt |
|
drainage channel |
The whole part where the screws are installed. |
A screenshot of tagging images in Azure Percept Studio.
After tagging only a few images, we started our first training iteration to take advantage of the AI-assisted object detection for tagging further images. Also, this way, we were seeing a steady progress of our model and how it performed. This is the result of our first iteration:
A screenshot of the result of our first iteration.
We didn’t have a lot of data yet, so the results were far from what we were hoping for, but it was a start. We proceeded to push our first iteration to the Azure Percept Dev Kit to see our model in action.
A picture of how tightened screws were detected as empty holes.
The picture shows how the tightened screws were detected as empty holes. However, the loose bolts on the top are not being detected at all. At this point, we decided to gather more training data and enhance our model. Additionally, we were getting rid of the “drainage channel” tag, focusing on the remaining tags.
To further progress with the model development, we came up with the following development cycle to make increments and achieve better results.
A picture of the development cycle to make increments and achieve better results.
After just five iterations, we had gained a lot more training data for the model. We were not only using images from Azure Percept, but also taking pictures with our cell phone cameras, modifying the lighting, changing the background, and inserting or removing the screws in various ways. In this way, we got different scenarios to prevent our model from being overfitted. The results were promising:
A screenshot of the result after 5 iterations.
There was a significant increase in loose bolt and empty hole detection, however there were still some issues left to be addressed. Some parts were still hardly detected at all, and we were missing some training data for loose bolts. Seven iterations later we were finally satisfied with the results. Everything we trained the model for was now detected with an average certainty of over 70%. Also, the lighting was less of an issue for detection.
Final Thoughts
Azure Percept offers a quick and convenient way to bring AI based visual and audio solutions to the edge without the need to go into the math behind the models to be trained. This enables a much faster time to market. In just about one day, it is possible to build a solution like the one presented above from scratch.
Even though the solution is not yet ready for production, it is a solid starting point to integrate AI-based solutions in your IoT solution. With the results of Azure Percept being brought to the cloud, the possibilities for further enhancements are huge. The next step could be notifications about changes in a screw’s state to help avoid further damage to the affected assembly and therefore prevent additional costs and injuries. No matter if used for quality control of production or to monitor the facility, adding a notification system will make sure the information is delivered to the right people at the right time. Using services like Logic Apps can help with the integration of various communication channels, like email or even a Microsoft Teams channel for the facility management.
To learn more about Azure Percept, visit these resources:
- Learn about Azure Percept
- Learn about Azure Percept Technical Documentation
- Learn more about our customer and partner stories
Microsoft Build 2022 Key Sessions
- Microsoft Build Into Focus: Preparing for the metaverse
- Embrace digital transformation at the edge with Azure Percept
GTC Spring 2022 Key Sessions
- Session: Transforming AI and ML at the edge with Microsoft and NVIDIA
- Embrace digital transformation at the edge with Azure Percept