With the growing presence of intelligence at the edge, it is critical to support a high-level of security to prevent malicious attacks on edge devices. Therefore, it is essential to build a robust and resilient update story to enable devices to be easily patched. Microsoft Azure Percept delivers a leading over-the-air update experience using services and tools native to Azure.
For Azure customers and partners, secure update is one of the first device management functionalities they want to adopt. In Azure Percept, the update stack is core to the devices and experiences. In this article, find out how to build a robust and resilient update story for your IoT and edge AI devices by harnessing the power of Azure and the recently announced Device Update for IoT Hub.
Understanding the Azure Percept update architecture
Azure Percept uses an atomic A/B image update to update the host operating system (OS) and firmware (FW) using Device Update for IoT Hub, which is a comprehensive platform to operate, maintain, and update IoT devices at scale. The business logic and AI models run as containers on the device and are updated using Azure IoT Edge.
Azure IoT Edge moves cloud analytics and custom business logic to devices so that your organization can focus on business insights instead of data management. Scale out your IoT solution by packaging your business logic into standard containers, which can be deployed to any of your devices and monitored from the cloud.
Device Update for IoT Hub empowers users to rapidly respond to security threats and to deploy new features to meet business objectives without incurring the additional development and maintenance costs of building their own update platform. Device Update for IoT Hub implements robust security measures, as well as rich management controls and reporting to help ensure customers stay in control of the update process. Device Update for IoT Hub is now available in preview in the Western U.S., Asia, and Europe.
Exploring the Azure Percept infrastructure
The Azure Percept engineering team used the power of Azure to develop the ideal infrastructure to build, test, and publish updates.
Azure DevOps and Build Pipeline
Azure Percept update artifacts are created through Azure Pipelines in Azure DevOps. Pipelines enable the engineering team to continuously build, test, and deploy to any platform and cloud. With Pipelines, we have automated the Azure Percept builds and deployments, empowering the engineering team to spend less time on manual tasks and more time being creative. The pipeline has ARM64 and X64 servers running the corresponding flavors of the Ubuntu Operating System. Azure DevOps Pipeline supplies key tasks, including the authoring of the Bash script to carry out the build task. The output artifact of the Azure DevOps pipeline includes the base image file (RAW.XZ), update image file (SWU), and the import manifest file (JSON).
Automated VM testing
An update issue can lead to a loss of devices, which increases the maintenance cost of recovering bricked units and significantly impacts end-to-end operations. Which is why Azure Percept integrates automated VM testing using Azure Pipelines. Daily automated tests validate the update stack end to end. High-level steps include:
- A dedicated Windows Server Machine has a base version of Azure Percept VM Image. The Azure DevOps Pipeline automatically provisions this VM Image
- The build pipeline creates the images and update artifacts
- Azure DevOps Pipeline is used to automate the following steps of import, deployment, and validation:
- The Device Update Import API imports the update file (SWU file) and import manifest file
- Upon completion, the Device Update Deployment API schedules a deployment to the VM device
- The VM device runs the Device Update Agent as a service. Once the deployment is scheduled, the device receives notification of the update, downloads and installs the update, and reboots to complete the update flow
- The Device Update Deployment API tracks the status of deployment
Signing image
To help improve the security of the update process, the SWU manifest file is signed with a dedicated key for Azure Percept update images. Manifest file creation and key signing are managed through an Azure DevOps Pipeline task.
Understanding device-side integrations
Device-side integrations, including partition layouts, agent integrations, and installer setups, are essential for enabling a resilient and robust update stack for IoT and edge AI devices.
Atomic image update
Unlike laptops or smartphones, IoT devices typically don’t have a user present to support the update process. Updates that break devices or cause operational failures can cause device downtime or other failures, leading to operational disruptions, data loss, and high replacement costs. To reduce these risks, the Azure Percept team implemented atomic image updates. Atomic updates are full image updates that target an entire device across its firmware and operating system. Instead of updating a singular component, atomic updates target all device components. The atomicity of the update minimizes the risk of an update breaking a device. Atomic updates are ideal for smaller edge devices in environments where bandwidth is not limited. To reduce the update size and accommodate larger edge devices and those in bandwidth-constrained environments, the Azure Percept team is engaging with partners and investigating feature developments.
Dual A/B partition
To deliver image updates, Azure Percept uses a dual-image partition, which supports rollback functionality and minimizes the risk of impacts to devices in operation. If an update fails on a device, rollback allows the device to return to its pre-update state.
Figure 2: Reference partition layout for the Azure Percept device
Each update is downloaded to the data partition, where it is processed and applied to the secondary partition. The device reboots into the secondary partition after the update. If the update fails, it has the capability to rollback into the primary partition.
Running update agent as an auto-start service
Azure Percept devices have integrated the Device Update for IoT Hub agent to run as an auto-start service (adu-agent.service). The service reads the connection string information from the Azure IoT Edge configuration file (/etc/iotedge/config.yaml). The Azure Percept device is initialized via automatic device provisioning or the Onboarding Experience. This initialization phase sets the Device Provisioning Service (DPS) scope ID or the Azure IoT Hub connection string in the Azure IoT Edge config.yaml file, depending on the initialization mechanism. The service uses the information in /etc/iotedge/config.yaml to connect the digital instance of the device. The Device Update for IoT Hub agent communicates with the Device Update for IoT Hub service over the IoT Hub device twin.
SWUpdate
To process and install the update artifacts, Azure Percept uses SWUpdate, which is an installer implementation supported by leading Linux developers. This project is open source on GitHub and is actively maintained by the community. SWUpdate is a manifest-driven update architecture; the manifest declares the update payload and the install method. SWUpdate provides options to overwrite an entire partition with a raw disk and write a binary file to a specific disk offset. While SWUpdate has proven itself to be the best choice for Azure Percept, other installers may be used, depending on the specific scenario requirements.
Figure 3: Sample manifest file showing all metadata associated with a device update for Azure Percept
Import phase
The import phase allows the device builder to import the update artifacts (update file and import manifest) to the Device Update for IoT Hub service. Once the device builder schedules a release build, Azure Pipelines creates the image file (RAW.XZ), update file (SWU), and import manifest (JSON). The image file (RAW.XZ) can be flashed to reset a device or to perform an offline USB-driven update. The device builder can use the Device Update for IoT Hub UX or API to import the update file and import manifest into the Device Update for IoT Hub service.
Figure 4: Import phase showing how the Azure Percept team builds and imports the latest update artifacts
Deployment phase
The deployment phase allows the device builder and solution operator to deploy updates to a group of Azure Percept devices. To deploy an update to an Azure Percept device, a user must:
- Create device groups using the UX or APIs.
- Add devices to the appropriate device groups.
- Set a deployment start time (UTC) for each device group.
Once the user actions are completed, the Device Update for IoT Hub service communicates with the Azure Percept device via IoT Hub device twin messages. The Delivery Optimization Agent downloads the payload and the SWUpdate Agent installs the update. The Device Update for IoT Hub agent reboots the device to finalize the install. To complete the feedback loop, the device reports the update status back to Azure IoT Hub.
Figure 5: Deployment phase showing how an IoT solution operator can deploy the latest Azure Percept updates to their devices
What’s next?
Microsoft and Azure equip you with the right tools and technology to build a resilient update stack with added security to help safeguard your edge AI devices. To use an Azure Percept device and see the update stack in action, join the Public Preview. For more information on the Azure Percept program and for developer resources, check out Microsoft Docs.
