Solve common challenges with Azure Percept: Inspirations from Microsoft Hackathon
Published Dec 15 2021 02:17 PM 3,974 Views

Creativity, innovation, and forward thinking are all part of a never-ending journey and exploration of what's possible. We've seen this time and time again as innovators and tinkerers have dreamed up new ways to use the Azure Percept platform and its development kit to enable intelligent edge AI solutions. And the best part: this innovation isn't slowing down any time soon.


I recently shared some of the amazing ways Azure Percept is being used to create ingenious solutions across industries like manufacturing, retail, smart cities, smart buildings, and transportation. Fast forward just a few months, a number of our employees further explored the capabilities of Azure Percept during a recent Microsoft Global Hackathon.


In the Microsoft Hackathon, Microsoft employees completed numerous hacks with Azure Percept and brainstormed ways to create solutions for real-world challenges ranging from pedestrian safety to supply-chain efficiency. They used the Azure Percept development kit's built-in hardware accelerators, which integrate seamlessly with Azure AI and Azure IoT services, to experiment with what's possible. Below are a few of these real-world solutions that target uses cases across five different verticals: transportation, healthcare, telecommunications, AI for good, and public safety.


Transportation: Protecting against train incidents

Credit to LeAnn Huang


Determined to prevent devastating train derailments, one team developed a potential solution that uses Azure Percept Vision to check for potential obstacles on tracks. The solution would use cameras mounted above train tunnel entrances and run anomaly detection checks. If a hazard is detected on the tracks, the Azure Percept system would send a signal to activate warning lights and notify the train's driver to brake. Areas outside of the camera's range could be augmented by using drones to detect anomalies.


"The impact is huge, and more lives can be saved if we can start to apply edge AI solutions to address the train derailment issue," said LeAnn Huang, a senior software engineer who worked on this solution.


In cases where there may not be time to stop a collision between a train and a detected object, the system would deploy a large airbag to reduce the crash force to the train and passengers in addition to directing the driver to brake. Such airbag devices can reduce the force level of a collision by 30 percent, according to Huang's simulation. The system also could call emergency services if Azure Percept Audio detected the sounds of a collision. At the same time, information related to an incident could be sent automatically through IoT Hub and Azure Video Analyzer to railroad controllers, Huang said.


Check out the protecting against train incidents demo:


Healthcare: Using Edge AI to improve hand-washing compliance

Credit to Oz Hernandez


Because infections acquired at hospitals affect thousands of patients annually, one solution focuses on improving hand hygiene compliance in health care facilities and even restaurants. The team's solution uses Azure Percept and computer vision to detect whether hands are present over the sink; Azure Sphere, paired with a passive infrared (PIR) sensor, then measures soap usage.


This type of solution could help cut down on the spread of pathogens on hands, as a Centers for Disease Control study found healthcare providers wash their hands less than half of the time they should. Similarly, handwashing by employees in the food services industry tends to be lower than expected.


"Hand-washing compliance can help save lives, and this solution could help healthcare facilities to increase and track hand-washing compliance," said Oz Hernandez, a Microsoft senior program manager on the team that created this solution, adding Azure Percept reduced the solution's development time. "You can spend more time focusing on solving a business problem instead of spending too much time learning the different pieces of the technology stack."


Check out the using Edge AI to improve hand-washing compliance demo:


Telecommunications: Leveraging 5G to empower edge AI

Credit to Karl Davies-Barrett


Tracking medical tools during surgery and making sure they're all accounted for before a procedure wraps up can help surgeons focus on their patients and reduce medical mistakes. This team showed what can be accomplished by an edge AI solution that uses Azure Percept in conjunction with IoT, live video analytics, private 5G networks, and private multi-access edge compute (PMEC). This innovative combination of technologies can be applied for other uses as well.


"Live video analytics is a great way to bring quick value to any business, and the combination with 5G means you can get your solution in places where Wi-Fi would not be possible," said Karl Davies-Barrett, a senior cloud solution architect. "That, coupled with high speed and low latency is allowing new solutions to be brought to market around smart cities, smart stadiums, airport security, etc."


The idea is to create compelling 5G solutions to use cases that can be offered in an end-to-end package. Such a solution could pair the use of Azure Percept with Azure Stack HCI and technology from Microsoft partners such as Nvidia, in addition to using and being shared on Azure architecture with the help of the Azure for Operators team.


Check out the leveraging 5G to empower edge AI demo:


AI for Good: Transcribing sign language

Credit to Karl Davies-Barrett


Using Azure Percept's dev kit to make everyday life more accessible for people with hearing impairments or other disabilities is also a possibility, as one developer has shown. The developer trained an AI vision model to translate letters and words signed in American Sign Language into words displayed on the screen.


Sign Language Live Long and Prospoer.png


With that demonstrated, it should be possible to translate speech into ASL so that anyone could communicate through sign language. This solution uses Azure Percept with Azure Cognitive Services to train a machine learning model deployed to the dev kit. The translated signs are displayed using PowerApps.


Especially for virtual healthcare visits, this real-time translation back and forth between speech and American Sign Language can open up better communication—and better diagnoses and treatment—for care providers and patients.


"The ability to connect a doctor remotely to a patient is exciting, allowing them to reach deprived areas where healthcare is too expensive or under-served," said Davies-Barrett, who also worked on this hack. "AI will democratize healthcare, and I think Azure Percept can play a strong role in not only prototyping solutions but also in producing them very quickly and cheaply."


Safety: Touchless entry using a biometric access control system

Credit to Nirnay Bansal


Touchless entry technology isn't new, but this solution demonstrates how the computer vision capabilities of Azure Percept can be harnessed to power a simple access control system.




This can be used in facilities ranging from workplaces to schools to entertainment venues with restricted areas. A demonstration of this proposed solution showed how feeding photos of a person into an Azure Percept AI model could accurately identify that person and allow them access to a space.


Nirnay Bansal, an engineering team lead, said he created the Microsoft Custom Vision model in a few minutes. "My hack project used face detection, which is the simplest use case that could be used in schools, border protection, offices, and hospitals," he said.


The same vision detection and analytics used to identify people also can be used instead for object detection, including to detect if medication is being consumed by patients in hospitals or senior care facilities, or to help track surgical tools to make sure they don't get lost during surgeries or other medical procedures, Bansal said.


Join in and start innovating with Azure Percept

Our customers, partners, IoT tech community, and internal teams have shown that just about any solution idea can be turned into reality with Azure Percept. Its development kit is designed for rapid prototyping, creating quick solutions to everyday challenges without requiring extensive coding, and we know we've only seen the beginning of the ways Azure Percept can be used.


Have an idea you want to test out? You can get started by learning more about Azure Percept and ordering a development kit today. We can't wait to see what creative and cutting-edge ways you think of to deploy edge AI capabilities with Azure Percept!


Resources for learning more about Azure Percept

Version history
Last update:
‎Dec 18 2021 10:32 AM
Updated by: