Continuous quality control in manufacturing using Azure Percept
Published Jul 05 2022 08:00 AM 2,238 Views
Copper Contributor



In manufacturing as well as other industries, safety, like logistics, needs special attention. Not only can a lack of safety mechanisms result in downtime and costly maintenance, but also, even worse, in physical harm to the staff operating the facilities. One of these safety obstacles is caused by loose screws and bolts. Constant vibrations, corrosion, and mistakes made during the assembly are the main drivers for a declining preload force of screws. Considering the harsh environment present in manufacturing and logistics, it’s only a matter of time until the first screw connections will fail.


Therefore, CANCOM developed a solution detecting loose screws and bolts using Azure Percept, while participating in the Azure Percept Bootcamp. This blog post shows how the solution works using the example of a drainage channel.


Solution architecture


Our continuous quality control solution monitors the state of screwed assemblies using Azure Percept. The following is the architecture for the solution.




Core components


The core components of the solution are:




Azure Percept

Captures the data stream from the vision module of the Azure Percept Dev Kit and uses a pre-built AI model to detect the state of screwed assemblies.

IoT Hub

Handles the communication with the IoT Hub in the cloud.

Cognitive Services

Service for training the AI model.

Azure Percept Studio

Management portal for an easy development and management experience of the Azure Percept solution.

IoT Hub

Receives the data from the modules running on the edge and enables the management of all workloads on the edge.

Stream Analytics

Processes the raw data from the IoT Hub and stores it in the Azure SQL Database.

Azure SQL Database

Storage to persist the data collected.

Power BI

Visualizes the data from the Azure SQL Database. Note that it is also possible to either skip the database or add a second output from the Stream Analytics service directly to Power BI to achieve dashboards consuming stream data and delivering near real-time insights on the current state of the monitored facilities.


Solution Implementation


For simulation of a channel drainage, a demo setup as shown below was used.




After setting up the Azure Percept Dev Kit, the other components can be deployed and configured.

First, the database is created with a table of the desired telemetry data schema. Then, an IoT Hub with a dedicated consumer group for the Stream Analytics service is needed. After that, the IoT Hub can be configured as a stream input in the Stream Analytics service. The Stream Analytics service uses its query to format the raw data of the IoT Hub and saves it in the database configured as output.


Training the model


As previously mentioned, the training of the AI-model was done using Azure Cognitive Services. We created a custom vision model and gathered our first images using the Image capture functionality in Azure Percept Studio.

We then proceeded to tag the images with these tags:




bolt ok

A fully tightened screw or bolt

bolt loose

A screw or bolt that came loose. Usually where the thread of the screw was visible


A screw that is not inside a socket

empty screw hole

An empty hole missing a screw or bolt

drainage channel

The whole part where the screws are installed.



A screenshot of tagging images in Azure Percept Studio.


After tagging only a few images, we started our first training iteration to take advantage of the AI-assisted object detection for tagging further images. Also, this way, we were seeing a steady progress of our model and how it performed. This is the result of our first iteration:


A screenshot of the result of our first iteration.


We didn’t have a lot of data yet, so the results were far from what we were hoping for, but it was a start. We proceeded to push our first iteration to the Azure Percept Dev Kit to see our model in action.


A picture of how tightened screws were detected as empty holes.


The picture shows how the tightened screws were detected as empty holes. However, the loose bolts on the top are not being detected at all. At this point, we decided to gather more training data and enhance our model. Additionally, we were getting rid of the “drainage channel” tag, focusing on the remaining tags.

To further progress with the model development, we came up with the following development cycle to make increments and achieve better results. 



A picture of the development cycle to make increments and achieve better results.


After just five iterations, we had gained a lot more training data for the model. We were not only using images from Azure Percept, but also taking pictures with our cell phone cameras, modifying the lighting, changing the background, and inserting or removing the screws in various ways. In this way, we got different scenarios to prevent our model from being overfitted. The results were promising:



A screenshot of the result after 5 iterations.


There was a significant increase in loose bolt and empty hole detection, however there were still some issues left to be addressed. Some parts were still hardly detected at all, and we were missing some training data for loose bolts. Seven iterations later we were finally satisfied with the results. Everything we trained the model for was now detected with an average certainty of over 70%. Also, the lighting was less of an issue for detection.




Final Thoughts


Azure Percept offers a quick and convenient way to bring AI based visual and audio solutions to the edge without the need to go into the math behind the models to be trained. This enables a much faster time to market. In just about one day, it is possible to build a solution like the one presented above from scratch.


Even though the solution is not yet ready for production, it is a solid starting point to integrate AI-based solutions in your IoT solution. With the results of Azure Percept being brought to the cloud, the possibilities for further enhancements are huge. The next step could be notifications about changes in a screw’s state to help avoid further damage to the affected assembly and therefore prevent additional costs and injuries. No matter if used for quality control of production or to monitor the facility, adding a notification system will make sure the information is delivered to the right people at the right time. Using services like Logic Apps can help with the integration of various communication channels, like email or even a Microsoft Teams channel for the facility management.


To learn more about Azure Percept, visit these resources:  


Microsoft Build 2022 Key Sessions  


GTC Spring 2022 Key Sessions  


Version history
Last update:
‎Jun 29 2022 11:30 AM
Updated by: