We are excited to share with you some of the latest features that we have added to Azure AI Video Indexer (VI), the video analysis solution that helps you to extract insights from your video and audio files. Here are some of the highlights:
- Edge version of Azure VI, enabled by Arc: You can now use Azure VI on your own edge devices, without sending your video data to the cloud. This will allow adhering to local regulation, cost saving, efficiency and security constrains, while still enjoying the same functionalities that Azure VI offers on the cloud.
- Bring-your-own AI feature: You can now use models from Hugging Face, Azure Florence, or any other AI model and connect them to VI insights. This will allow you to customize and enhance your video analysis with your own models, and to benefit from a single joint VI experience.
- Enrich your video metadata and search experience: You can now add custom tags and free text as video metadata for any specific video in your account. This will help you to annotate your videos with any information that is relevant to your business. You can also search for videos based on the custom tags and free text that you have added to them.
- Features improving the efficiency of customized people model (subjected to limited access features program)
All features in this announcement are delivered as preview features.
Now with more details:
Announcement #1: Running VI AIs on Edge
The first cloud-to-edge managed solution for video AI
As part of Azure’s adaptive cloud approach mentioned by Satya, we are thrilled to announce that we have released an edge preview version of Azure Video Indexer (VI), enabled by Arc. This version allows you to use the same managed video analysis solution both on cloud and edge, depending on your needs. In this adaptive version, we assume network connectivity is allowed, but the entire AI processing is on your edge appliance while VI is deployed on AKS cluster and the edge device has enough compute. Review Azure Arc Jumpstart to learn how to use VI enabled by Arc on AKS Edge Essentials single node deployment (in preview). The preview version is available starting December 15, 2023. Register here to get access.
Which business problem do we solve with Hybrid Azure AI Video Indexer approach?
Let's use this example - Contoso Supermarket is a retail company that operates worldwide. Some of its sites have CCTV cameras and they would like to use them to optimize their business performance. One of their goals is to detect cars in the parking lot in front of the stores and analyze if there is a correlation between the available parking and the number of orders. This way, they can adjust their inventory, staffing, and marketing strategies accordingly.
However, sending all the video footage from the cameras to the cloud for vehicle analysis would be costly, slow, and not always align with local regulations. Each supermarket has many hours of video recorded every day, which would consume a lot of bandwidth and storage. The video analysis would also take a long time to complete. Moreover, the video data may contain sensitive information about the customers and the employees, which may violate some local regulations or privacy policies.
That is why Contoso Supermarket decided to use the adaptive and hybrid video analysis solution with Azure VI. This solution allows them to run the AI processing on their own edge devices, without sending the video data to the cloud, but only share the results of the indexing with the regional operator that works and manages the region through the cloud.
Read more about the new version in this blog by @shemers .
Image 1: Example of Video Indexer AI pipeline running on edge. Basic video and audio AIs will be available on edge, in addition to the cloud. Custom AI models can be integrated to the insights as well
Announcement #2: Bring external AI models
Merge your AI models with VI AI pipeline and enjoy a unified experience
Microsoft Azure AI delivers the state-of-the-art AIs, but fine-tuning is required for specific vision and audio analysis tasks, such as license plate detection, tattoo detection, special signs in video, specific uniforms/masks and more. Today we announce the bring-your-own AI feature, where you can use models from Hugging Face, Azure Florence, or any other AI model and connect it to VI insights. The models are hosted out of VI. This way, the video analyst can benefit from a single joint experience and can both see the insights on the portal and also search for them. See below how we integrate a custom model of car types with VI insights:
Image 2: example of integration of custom car models AI
Announcement #3: Improve VI search experience
With custom tags, custom metadate, search for objects, search by file name
We are excited to announce that we have released some new features for Azure VI that will help you to enrich your video metadata and improve your video search experience. These features are:
- Custom tags and free text per video: You can now add custom tags and free text as video metadata for any specific video in your account. This will allow you to categorize and annotate your videos with any information that is relevant to you and your business. For example, you can add tags such as “product demo”, “customer testimonial”, or “internal training” to your videos, or you can add free text such as “This video shows how to use our new feature X” or “This video was recorded at our annual conference in Y”. You can add custom tags and free text after the video has been indexed.
- Search based on custom tags and free text: You can now search for videos based on the custom tags and free text that you have added to them. This will enable you to find the videos that match your criteria more easily and quickly. You can search based on custom tags and free text in all supported languages, and you can combine them with other search filters such as keywords, faces, labels, emotions, etc. You can also use the advanced search syntax to perform more complex queries. For example, you can search for videos that have the tag “product demo” and the free text “feature X”
- Search by file name: you can search for a specific title of a file you uploaded.
Image 3: user can add custom video tags under the video player
Announcement #4: Features improving the efficiency of customized people model
Limited access features as these features related to people model which is available as part of a gating program for limited access features in Azure AI Video Indexer
We are also happy to announce that we have released new feature set in preview to improve the efficiency and accuracy of customized People model, a gated AI model that allows you to train your own model to recognize specific people in your videos. These features are:
- Indication on the quality of people model: You can now get an indication on the quality of your customized People model (poor, fair, good) based on the number of images used for labeling. The more images you use to label a person, the higher the probability to identify the person correctly. For example, the probability to identify a person with 24 labeled images is higher than the probability to identify a person with 2 labeled images. You can see the number of images used for labeling each person in your customized People model page.
Image 4: Indication on the quality of people model
- Choose a custom people model as default: You can now choose a customized People model as default on VI account user level, avoiding repeating the selection of the model name every video upload. This will save you time and effort when you upload videos that need to be analyzed by your customized People model.
- Grouping of unknown people in the video: You can now see the unknown people in your videos grouped by their appearance similarity. This will help you to label the unknown people more easily and quickly, and to improve the accuracy of your customized People model. You can see the grouping of unknown people in your customization page > choose people and then navigate to the “unknown people” tab. This could e.g. help you to label a local celebrity or a local politician.
Image 5: cluster of a detected person face from the video library
- Search results with max confidence score for identified person name: You can now search for the name of an identified person and get the video results where the person appears, with the maximum confidence score. This helps you to decide what are the most relevant videos to explore. For example, you can search for “John Smith” and get the videos where John Smith is recognized by your customized People model, along with the confidence score for each video
Image 6: Search results with max confidence score for identified person name
We hope you enjoy these new features and we look forward to hearing your feedback.
Thank you for choosing Azure VI as your video analysis solution.
Explore Azure AI Video Indexer Today
- Use Azure Video Indexer website to access product website and get a free trial experience.
- Visit Azure Video Indexer Developer Portal to learn about our APIs.
- Search the Azure Video Indexer GitHub repository
- Review our product documentation.
- Get to know the recent features using Azure Video Indexer release notes.
- Use Stack overflow community for technical questions.
- To report an issue with Azure Video Indexer (paid account customers) Go to Azure portal Help + support. Create a new support request. Your request will be tracked within SLA.
- Read our recent blogs in Azure Tech Community.