Blog Post

AI - Azure AI services Blog
5 MIN READ

Computer Vision for spatial analysis at the Edge

AdinaTru's avatar
AdinaTru
Icon for Microsoft rankMicrosoft
Sep 22, 2020

Today businesses use manual processes to understand their physical spaces and meet business requirements such as maximizing revenue for store layouts, compliance, worker safety in manufacturing plants, and more. These manual processes are occurring infrequently and through inefficient methods where employees manually count people entering stores and visually monitor social distancing requirements.

 

Computer Vision, an Azure Cognitive Service, is introducing the spatial analysis feature to meet the needs of businesses across a variety of industries. Spatial analysis uses Computer Vision AI on real-time video and offers the ability to understand people’s movements in a physical space, significantly increasing efficiency and resolution of customer data. Equipped with this net new knowledge, employee time can be spent on high value experiences, to maximize return on investment, including efficient product placement, shelf stocking, sanitation, and focusing on improving the customer experience.

 

With AI Edge container support, spatial analysis provides the flexibility to solve current challenges around data control, privacy, and network intensive video AI workloads. Spatial analysis brings the power of the Intelligent Edge and the Azure cloud to create powerful business applications at scale.

 

Spatial analysis runs on the Intelligent Edge

Understanding the physical world using video is very complex, requiring deployments of video pipelines, hardware, AI models, and processing insights, all of which need to be connected into the existing infrastructure and aggregated into useful views. Organizations want simple deployments that can be hooked into their existing cameras as well as using new cameras that can be quickly installed to get immediate business value. The goal of this post is to demonstrate how enterprise applications can use the spatial analysis container on the edge, and how to use the Azure cloud to create powerful business applications.

 

Customer Story

RXR Realty needed a way to integrate new safety measures for tenants after its buildings reopened for business during the COVID-19 pandemic. The RxWell™ solution built by RXR is a comprehensive, public-health–based, data-driven program that merges physical and digital assets to help keep employees informed and supported during the “new abnormal” and beyond. RXR leveraged the Computer Vision spatial analysis capabilities of Cognitive Services to count the number of people in a specific zone and calculate the distance between each person. RxWell™ securely runs the models at the edge in Docker containers on Azure Stack Edge hardware. The resulting AI insights are sent to the cloud, where they’re used for real-time alerting and historical trending. To learn more about the RxWell™ solution with spatial analysis see this article.

 

Spatial analysis container for Azure IoT Edge deployments

Computer Vision for spatial analysis is a collection of AI operations using AI models. These operations are the building blocks that enable scenarios including:

  • Counting people in a space for maximum occupancy
  • Understanding the distance between people for social distancing measures
  • Determining footfall such as in retail spaces
  • Understanding dwell time such as in front of a retail display or other designated location
  • Determining wait time in a queue
  • Determining when people are in a forbidden zone such as near industrial equipment
  • Determining trespassing in protected areas

This list portrays a few examples of how spatial analysis can be utilized in a variety of scenarios. In addition to these scenarios, real-time messaging can power customer workflows including Teams, Power Apps, Power Automate, and saved events can enable powerful reports with Power BI. 

 

Developers, enterprise solution providers, and customers with development capabilities can use these AI operations to build edge or cloud solutions, with no Machine Learning experience required. The ready-to-use, high-quality AI Models for spatial analysis operations are trained to understand people movement across a wide variety of scenarios, camera types, angles, and lighting conditions.

 

Spatial analysis operations implement a real-time video pipeline to connect to new and existing RTSP cameras, including Closed Circuit TV systems. The deployment of the spatial analysis container on edge devices is facilitated by Azure IoT Hub. You're in control of your edge hardware located on your premises. The recommended edge device is Azure Stack Edge with the Nvidia T4 GPU. Other edge devices may be used if an Nvidia T4 GPU is available.

 

Spatial analysis container deployment with Azure IoT 

You need to install the camera devices in your physical space following the manufacturer’s instructions. The edge device hosting the container needs to be able to connect to these cameras over the RTSP protocol and stream video. Camera placement is an important step in the deployment process. For general guidelines and specific recommendations for height, angle, and camera-to-focal-point-distance, see Camera placement article.

 

When video is streamed and processed by the spatial analysis AI models, the container emits AI Insight events about people’s movement which in turn are sent to Azure IoT Hub as IoT telemetry. From IoT Hub you can create various routes to other Azure services and build your business solution. You may also decide to process the AI Insights locally on the edge device.

 

Live Video Analytics with spatial analysis

The spatial analysis container container can be deployed side by side with the Live Video Analytics container. In this configuration, the Live Video Analytics container will stream the live video from the RTSP cameras, and it will invoke spatial analysis for AI processing. The Live Video Analytics and spatial analysis containers running on Azure IoT Edge can be configured to enable rich video analysis and recording of video clips locally or to Azure Blob Storage.

 

Azure IoT deployment for Live Video Analytics and spatial analysis containers

 

Building business applications with spatial analysis AI insights

You can build various business applications using the spatial analysis AI insights for people movement. Follow these instructions to deploy a sample Azure Web Application that presents a live view of people counting events in a physical space. The AI insights events emitted by the spatial analysis container are being sent to Azure IoT Hub and subsequently to the Web Application, which in this case implements a visualization with a chart that updates the people count in real time as shown below. You can further modify this app with other spatial analysis operations and make modifications based on the event output of the container. When it comes to business decisions centered around social distancing requirements, determining store layouts, adding team members to reduce wait times, or improving customer satisfaction, spatial analysis makes it easy to gain these valuable insights.

 

 

Person count visualization in a web application

 

Responsible AI & Innovation

Microsoft is releasing Computer Vision for spatial analysis together with responsible deployment guidance  grounded in user and societal research.

Microsoft developed the Responsible Deployment recommendations by applying many of the responsible innovation best practices in collaboration with customers to uncover deployment recommendations for spatial analysis in accordance with Microsoft Responsible AI Principles: fairness, reliability & safety, privacy & security, inclusiveness, transparency and human accountability.

Microsoft’s principled approach enables developers to build rich solutions while upholding the human dignity and the needs of everyone impacted by the technology.

 

Get Started Today

Learn more with Computer Vision for spatial analysis documentation

Follow the tutorial to Create a People Counting Web App

Get started with Azure Stack Edge 

Learn about Azure IoT Hub

 

Updated Sep 22, 2020
Version 1.0
  • hussnain_ahmed's avatar
    hussnain_ahmed
    Copper Contributor

    Is Tesla T4 GPU a recommendation or a mandatory requirement? Can we use Nvidia Jetson Series (TX2, Xavier NX etc.) as edge device for a single low frame rate video stream? 

     

  • hemantkamalakar hussnain_ahmed  Thanks for your interest in Spatial Analysis. We recommend NVIDIA T4 for production deployments. You may use a CUDA Compute Capable devices 6.0 or higher for testing ( e.g.: NVIDIA 1080Ti or 2080Ti ). 

     

     

  • hemantkamalakar's avatar
    hemantkamalakar
    Copper Contributor

    I would also like to run the Spatial Analytics container on nvidia Jetson Nano dev kit - 4GB.

  • hussnain_ahmed's avatar
    hussnain_ahmed
    Copper Contributor

    jfilcik recent announcement of Azure Percept, capable of running Spatial Analysis suites edge environments much better. Is Azure Percept DK good for production deployments or as a dev kit, it is only meant for prototyping? Secondly, do you have any roadmap for adding intel Movidius Myriad x devices to the list of recommended devices for the spatial analysis? The price of NVIDIA T4 based hardware is heavy on the business case for spatial analysis applications. Also, such servers are not suited to many edge environments.

  • AsaBurrows's avatar
    AsaBurrows
    Copper Contributor

    Hi, are there any tech forums specifically for Azure IotEdge/Spatial Analysis. I've been using it but have some challenges with it and could do with some community support?