Transform your Field Service Management with Azure Percept and HoloLens

Sargithan_Senthil · ‎Feb 17 2022

Microsoft Technology Centers has partnered with Kagool to demonstrate an end-to end field service management solution combining Azure Percept DK, AI, IoT, Cognitive Services and Mixed Reality with a physical Lego model.

Using Azure Digital twins, Mixed Reality and Azure Percept DK we digitalized the physical Lego Excavator into a digital presence, combining real-time telemetries and integrating a full end-to-end field service management solution. Real-time performance monitoring can be analyzed globally from a central location allowing you to reduce downtime and the cost of maintenance, optimize performance, reduce TCO and extend the life of your assets. Trend analysis enables intelligent servicing and predictive maintenance, allowing you to take preventative action before failures occur.

In this article our focus is to demonstrate how you can use the Azure Percept DK to integrate an end-to-end field service management solution and overlay the telemetries on a Digital Twin using Mixed Reality.

Overview of the solution

The architecture diagram below covers all the services we have used for implementing the field service management solution. The data which is being gathered by Azure Percept DK is being sent to IoT Hub where the Azure function is being triggered automatically (Event grid) to update the properties in Azure Digital Twins (ADT). In ADT, we rebuilt the global Lego asset deployments in a digital replica showing you exactly where each asset is deployed. Unity (Game Engine) will retrieve the properties based on custom queries sent to ADT using REST APIs and overlay telemetries on a 3D model.

Below is a video on how the digital twin responds to the real-time Azure IoT ingested telemetries (as viewed from HoloLens 2 device):

Implementation of the Solution

For you to get started, you will need to complete some prerequisites. You will need to set up your Azure Percept DK, create and configure an IoT Hub resource, train an AI model using Custom vision and lastly, set up your Function app instance to be able to move the data between services. If you need any help getting started, please look at the following resources:

Prerequisites:

Set up the Azure Percept DK

IoT Hub instance

Custom Vision instance

Function app instance

Azure digital twin instance

Unity project configured using Mixed Reality Toolkit

Steps of implementation:

Connect Azure Percept DK to IoT Hub. IoT Hub acts as a gateway for the IoT devices by collecting and then distributing the data accordingly. We have connected Azure Percept DK to IoT hub using the provided wizard. Below, we have attached a screenshot after configuring the device using the default settings:

Create and train an AI model using Custom Vision and deploy it on the device using Azure Percept Studio. As an example, we have trained an AI object classification model to detect the color of different marbles as shown in the image below:

Create an Azure Digital Twin instance and define the endpoints. We have used a layered approach by grouping each device based on its location (Continent – Region – City – Lego Asset). Example:

Configure an event grid-based Azure Function. This will automatically update the Azure Digital Twin properties by the telemetry received by IoT hub.

Deploy your function to the Function App instance. We have used Visual Studio to write the code required for the function to run and deployed it using the publish feature as shown in the screenshot below:

Go back to IoT hub, navigate to Events tab, create a new event grid schema subscription and select the previously created Azure Function as an endpoint. This is to ensure that all incoming messages are being routed to the right endpoint. After creating the event route, the main page should look like the screenshot below:

Configure your Unity Project to automatically update the data on the 3D Model. Using Azure Signal R, we can detect when a property is updated in Azure Digital Twins. Also, we used the powerful Azure Digital Twins APIs to send queries and retrieve metadata from ADT.

Configure the Unity project to display the ADT values. To enhance the user experience, we have displayed the Lego CAD model along with a few GUI elements:

Left Panel: Listing out all the globally deployed assets and their status’
Middle Panel: The real-time IoT telemetries that come from Azure Digital Twins
Right Panel: To initiate a work order back to SAP (ERP System) to replace a faulty component

Final thoughts

We have demonstrated to you how easy it is to integrate the Azure Percept DK with HoloLens. With this field service management solution, you can easily monitor your assets remotely from anywhere around the world and predict if a part is about to become defective to prevent any potential downtime or loss. This solution can be implemented in situations where humans cannot physically reach the asset location, or it would be a real danger for them to do so.

The benefits of this solution are many, we outline just a few below:

Remotely diagnose a performance problem anywhere in the world, create an inspection order, deploy maintenance personnel, and order replacement parts through seamless connectivity with your ERP system
Predict what failures are likely to occur and in what circumstances to intelligently set maintenance intervals, reduce downtime and save costs
Develop new commercial models based on pay-per-use through continuous ingestion storage and transformation of big data at high velocity
Centrally monitor performance against pre-set tolerances to identify problems, take preventative action and reduce downtime.