Resources for using ONNX Runtime AI on Jetson Embedded Devices in Education

In addition to my role as a Principal Cloud Advocate and Lead for IoT Advocacy at Microsoft, I act as a professor at the University of Houston where I teach a course focused on Cloud-Powered App Development. As part of this course, we focus on AI @ the Edge Scenarios backed by Microsoft Azure IoT Services.  To teach this concepts, we have chosen to target the very afforodable ($60 USD) NVIDIA Jetson Nano DevKit.  This experience is catalogued in-depth in a previous article on the Microsoft Educator Developer Blog. 

We find that student's most requested point of customization for AI @ Edge solutions is in the customization of the AI models themselves.  This can pose issues as there are various formats out there for developing computer vision based models, including: PyTorch, MXNet, Caffe, and Tensorflow.  Oftentimes, the tooling to develop these formats results in vendor lock-in, meaning that future applications of your model may be bound by limitations depending on tooling and runtime compatibility for building and executing the model in question.  

The Open Neural Network Exchange Format (ONNX) is a model standard/ format for exchanging deep learning models across platforms.  It's ability to be portable across model formats and even computer architectures makes it a prime candidate for AI model development without limitations.  It can even adapt to the presence of say GPU acceleration on a given computational platform to offer enhancement of your model at runtime, without any need to redevelop your model to take advantage of those optimizations.  Simply put, if you start with ONNX you can go anywhere and optimize without any extra effort.

Combining this fact with our target NVIDIA Jetson hardware, we can develop course content rooted in the development of ONNX based AI models to provide an open platform for students to build and experiment on, with the added benefit of GPU accelerated inference on low-cost embedded hardware.  This allows the students to apply concepts on a device that they can physically alter with the addition of cameras, microphones, or other sensors to aid in their solutioning.

As a basis for formal understanding of how ONNX works with NVIDIA GPUs, we recommend starting with Manash Goswami's presentation on the topic at the recent NVIDIA GTC 2020 conference (Note: that viewing this resource will require completing a free registration to NVIDIA's Developer Program Membership). 

After you have familiarized with the fundamentals, we recommend the following resources as hands-on lab or supplemental material for applying the concepts to run ONNX models on NVIDIA Jetson hardware for development of AI @ Edge solutions:

• Announcing ONNX Runtime Availability in the NVIDIA Jetson Zoo for High Performance Inferencing - NVIDIA Developer Blog 
• Integrate Azure with machine learning execution on the NVIDIA Jetson platform - Learn how to integrate Azure services with machine learning on the NVIDIA Jetson device using Python
• Intelligent-Video-Analytics-with-NVIDIA-Jetson-and-Microsoft-Azure - A repository demonstrating an end-to-end architecture for Intelligent Video Analytics using NVIDIA hardware with Microsoft Azure
• Azure IoT Edge Workshop: Visual Anomaly Detection over multiple cameras with NVIDIA Jetson Nano - In this workshop, you'll discover how to build a solution that can process several real-time video streams with an AI model on an NVIDIA Jetson Nano device, how to build your own AI model to detect custom anomalies and finally how to operate it remotely.

With these resources, we hope that you can employ the teaching of AI @ Edge scenarios without the fear of encountering the limitations inherent in less adaptable model formats.  ONNX based model development can ensure that your models are portable across architectures and adaptable to the existence of accelerators on compatible hardware.  For more information on the ONNX model format, be sure to check out https://www.onnxruntime.ai/ and for details on how to acquire NVIDIA Jetson embedded devices, check out this link.

Until next time,

Paul