Azure Percept is a platform of hardware and services that simplifies use of Azure AI technologies on the edge. The development kit comes with an intelligent camera, Azure Percept Vision, and it can also be extended with Azure Percept Audio, a linear microphone array. Azure Percept works out of the box with Azure services such as Cognitive Services, Machine Learning, Live Video Analytics, and others to deliver vision and audio insights in real time. Scenarios like object detection, spatial analytics, anomaly detection, keyword spotting, and others can easily be solved with use of pre-built Azure AI models for edge.
I build "Perceptmobile", an Azure Percept-powered obstacle avoidance LEGO Boost car, as a weekend project and in this post I will walk you through all steps how it was built.
In the standard LEGO Boost package you can find instructions on how to build 4 different models and one of them is M.T.R. 4 model, which I modified a bit to fit the needs of this project. Model was used as a base on top of which I placed Azure Percept with Azure Percept Vision camera. LEGO Boost package also comes with 3 cones that I took pictures of and trained the Custom Vision model. Custom vision models can easily be deployed to the Azure Percept via Azure Percept Studio, and you can easily test it with camera stream.
With NodeJS I made a small backend that is based on quickstart "Send telemetry from a device to an IoT hub and read it with a back-end application" and with use of Express I served results on the localhost. For the frontend application I used "Lego Boost Browser Application" which is a great React application for controlling LEGO Boost from the browser via Web Bluetooth API. With it you can easily connect to your LEGO Boost, and the application gives you a nice interface where you are not only able to control motors and other sensors, but you are also able to write different commands and programming logic.
In following video you can see full walkthrough how to build this project:
Quickstart: Send telemetry from a device to an IoT hub and read it with a back-end application (Node.js) can be found here.
Azure CLI commands used:
az iot hub show --query properties.eventHubEndpoints.events.endpoint --name {YourIoTHubName}
az iot hub show --query properties.eventHubEndpoints.events.path --name {YourIoTHubName}
az iot hub policy show --name service --query primaryKey --hub-name {YourIoTHubName}
Pictures used for model training and relevant code is available in GitHub repository.