Today businesses use manual processes to understand their physical spaces and meet business requirements such as maximizing revenue for store layouts, compliance, worker safety in manufacturing plants, and more. These manual processes are occurring infrequently and through inefficient methods where employees manually count people entering stores and visually monitor social distancing requirements.
Computer Vision, an Azure Cognitive Service, is introducing the spatial analysis feature to meet the needs of businesses across a variety of industries. Spatial analysis uses Computer Vision AI on real-time video and offers the ability to understand people’s movements in a physical space, significantly increasing efficiency and resolution of customer data. Equipped with this net new knowledge, employee time can be spent on high value experiences, to maximize return on investment, including efficient product placement, shelf stocking, sanitation, and focusing on improving the customer experience.
With AI Edge container support, spatial analysis provides the flexibility to solve current challenges around data control, privacy, and network intensive video AI workloads. Spatial analysis brings the power of the Intelligent Edge and the Azure cloud to create powerful business applications at scale.
Spatial analysis runs on the Intelligent Edge
Understanding the physical world using video is very complex, requiring deployments of video pipelines, hardware, AI models, and processing insights, all of which need to be connected into the existing infrastructure and aggregated into useful views. Organizations want simple deployments that can be hooked into their existing cameras as well as using new cameras that can be quickly installed to get immediate business value. The goal of this post is to demonstrate how enterprise applications can use the spatial analysis container on the edge, and how to use the Azure cloud to create powerful business applications.
RXR Realty needed a way to integrate new safety measures for tenants after its buildings reopened for business during the COVID-19 pandemic. The RxWell™ solution built by RXR is a comprehensive, public-health–based, data-driven program that merges physical and digital assets to help keep employees informed and supported during the “new abnormal” and beyond. RXR leveraged the Computer Vision spatial analysis capabilities of Cognitive Services to count the number of people in a specific zone and calculate the distance between each person. RxWell™ securely runs the models at the edge in Docker containers on Azure Stack Edge hardware. The resulting AI insights are sent to the cloud, where they’re used for real-time alerting and historical trending. To learn more about the RxWell™ solution with spatial analysis see this article.
Spatial analysis container for Azure IoT Edge deployments
Computer Vision for spatial analysis is a collection of AI operations using AI models. These operations are the building blocks that enable scenarios including:
Counting people in a space for maximum occupancy
Understanding the distance between people for social distancing measures
Determining footfall such as in retail spaces
Understanding dwell time such as in front of a retail display or other designated location
Determining wait time in a queue
Determining when people are in a forbidden zone such as near industrial equipment
Determining trespassing in protected areas
This list portrays a few examples of how spatial analysis can be utilized in a variety of scenarios. In addition to these scenarios, real-time messaging can power customer workflows including Teams, Power Apps, Power Automate, and saved events can enable powerful reports with Power BI.
Developers, enterprise solution providers, and customers with development capabilities can use these AI operations to build edge or cloud solutions, with no Machine Learning experience required. The ready-to-use, high-quality AI Models for spatial analysis operations are trained to understand people movement across a wide variety of scenarios, camera types, angles, and lighting conditions.
Spatial analysis operations implement a real-time video pipeline to connect to new and existing RTSP cameras, including Closed Circuit TV systems. The deployment of the spatial analysis container on edge devices is facilitated by Azure IoT Hub. You're in control of your edge hardware located on your premises. The recommended edge device is Azure Stack Edge with the Nvidia T4 GPU. Other edge devices may be used if an Nvidia T4 GPU is available.
Spatial analysis container deployment with Azure IoT
You need to install the camera devices in your physical space following the manufacturer’s instructions. The edge device hosting the container needs to be able to connect to these cameras over the RTSP protocol and stream video. Camera placement is an important step in the deployment process. For general guidelines and specific recommendations for height, angle, and camera-to-focal-point-distance, see Camera placement article.
When video is streamed and processed by the spatial analysis AI models, the container emits AI Insight events about people’s movement which in turn are sent to Azure IoT Hub as IoT telemetry. From IoT Hub you can create various routes to other Azure services and build your business solution. You may also decide to process the AI Insights locally on the edge device.
Live Video Analytics with spatial analysis
The spatial analysis container container can be deployed side by side with the Live Video Analytics container. In this configuration, the Live Video Analytics container will stream the live video from the RTSP cameras, and it will invoke spatial analysis for AI processing. The Live Video Analytics and spatial analysis containers running on Azure IoT Edge can be configured to enable rich video analysis and recording of video clips locally or to Azure Blob Storage.
Azure IoT deployment for Live Video Analytics and spatial analysis containers
Building business applications with spatial analysis AI insights
You can build various business applications using the spatial analysis AI insights for people movement. Follow these instructions to deploy a sample Azure Web Application that presents a live view of people counting events in a physical space. The AI insights events emitted by the spatial analysis container are being sent to Azure IoT Hub and subsequently to the Web Application, which in this case implements a visualization with a chart that updates the people count in real time as shown below. You can further modify this app with other spatial analysis operations and make modifications based on the event output of the container. When it comes to business decisions centered around social distancing requirements, determining store layouts, adding team members to reduce wait times, or improving customer satisfaction, spatial analysis makes it easy to gain these valuable insights.
Microsoft developed the Responsible Deployment recommendations by applying many of the responsible innovation best practices in collaboration with customers to uncover deployment recommendations for spatial analysis in accordance with Microsoft Responsible AI Principles: fairness, reliability & safety, privacy & security, inclusiveness, transparency and human accountability.
Microsoft’s principled approach enables developers to build rich solutions while upholding the human dignity and the needs of everyone impacted by the technology.