Machine Learning and Azure Sphere

The Internet of Things and Machine Learning

The Internet of Things (IoT) is arriving at pace, enabling new applications and business models across many consumer, enterprise, and industrial sectors. When you attach a “thing” to the Internet, you are connecting new data sources. which can be costly and resource intensive to store or upload.

That’s where machine learning (ML) comes to the fore. ML is excellent at processing noisy and complex sensor data to determine higher-level insights like “operating normally” versus “cooling unit failed” in a vibration sensor data stream, or “wake word detected” in an audio stream, or “items placed correctly” versus “item out of place” in a video stream.  

ML can be used on the IoT device itself, in the cloud, or in a combination of both.  ML on IoT devices brings the processing closer to the data generation. This has lots of benefits, including (a) internet connectivity is not relied upon in order for the higher-level states to be determined and actions to be taken on that basis, (b) fewer  resources are consumed for transmission of data to the cloud, and data that is not needed for the long-term can be immediately used and then deleted rather than stored, (c) it can enhance privacy, because the raw data might include more personal information (such as voices accidentally captured by an audio sensor designed to listen to machine behaviour), so if that data is processed and discarded locally, then there are fewer privacy risks.

Of course, there are also benefits to operating ML in the cloud. The availability of plentiful server resources means that cloud-based ML can be faster and use more complex models to achieve higher accuracy. Retraining of models, which is resource-intensive, can be quicker and redeployment of models can be immediate.

Hybrid designs are also possible, which involve ML on IoT devices and in the cloud. One example most of us will recognize is voice assistants—the wake word is recognized locally on the device, but then the voice data is streamed to a cloud service where language understanding and response generation is done.

Azure Sphere and ML

You can use Azure Sphere to build IoT devices while relying on Microsoft to stay on top of OS security threats, which means that you can focus on the actual application logic that defines your device, including ML where appropriate.

There are lots of advantages to using Azure Sphere for ML applications.

• First, Microsoft secures the device over its full lifetime. Since many devices requiring ML include sensors such as cameras or microphones, it is important to invest in ongoing security as this is a prerequisite for privacy—an insecure device cannot protect private data.
• Second, Azure Sphere provides a unique identity for the device and a certificate that encodes that identity for connection to cloud services, the result being that no other device can pretend to be this device. This is particularly important for ML applications because data that is falsely regarded as coming from a device can pollute training datasets, which degrades ML performance.
• Third, Azure Sphere’s application update service provides an easy way to update the ML model and ML runtime.
• Finally, ML models are valuable intellectual property, and the security of the Azure Sphere platform protects those models.

In a previous blog post, we described how Azure Sphere supports machine learning through “Tiny ML”, and detailed four different Tiny ML solutions that can run on the MT3620. Is this post, we will describe a demo that uses one of these ML solutions in combination with cloud ML service, showing how easy it is to build such hybrid ML solutions with Azure Sphere.

Recognizing faces: a hybrid ML example using Azure Sphere and Cognitive Services

To illustrate how Azure Sphere-based ML and cloud ML can work together, we’re going to use an example application which is recognising individuals. This could be used for scenarios like appliances which personalize their experience based on the user, or safety applications like only allowing authorized and trained individuals into potentially dangerous locations in an industrial building.

Face recognition is supported by an Azure Cognitive Service. It is quick and easy to train this cognitive service to recognize individuals. However, if this were to be used directly by an IoT device watching for a user, then that IoT device would have to upload pictures constantly, which is a poor design for many reasons including the bandwidth and cloud services costs, privacy implications, and energy consumption.

On the other hand, deploying this machine learning model to MT3620 presents difficulties. An appropriate ML model would need to be retrained for each individual user of each device, meaning that every IoT device would need a different model. Some ML models are large, and while various techniques can be used to compress them, ultimately this may not be achievable with high performance for some tasks.

The solution is to use the hybrid approach. In our face recognition example, we perform person detection on MT3620 using a pre-existing model provided by Mediatek as part of their Neuropilot-Micro framework. Then, only when a person is sensed do we need to use the Cognitive Service for face verification to determine identity.

Demo 

The following video shows the demo in action:

We built out this demo using the same person detection hardware and NeuroPilot Micro ML model documented by Mediatek. To use the Cognitive Services Face API, this webpage describes how to get started, and includes links to samples for that API.

Responsible AI

Finally, we should note that machine learning is a technology that it is particularly important to apply responsibly. ML-based products should be designed to protect principles such as being inclusive, human-centric, behaving understandably, and promoting fairness. At Microsoft, we take these responsibilities seriously for our own products, and we provide a range of resources to help our partners and customers navigate this space.