Create an AI-enabled Power App using an Object Detection Machine Learning model

 

 

This blog post is part 3 of a 4-part series. To catch up on previous posts, check out the links below:

• Part 1: Introduction to Power Apps
• Part 2: Training your Machine Learning Model

 

So, you’ve created your very own Object Detection Machine Learning model using AI Builder, and explored some Microsoft Learn modules to get comfortable using Power Apps. Now what? In part 3 of this series, we’ll build out our Power App – we’ll brainstorm what our app might look like, what functionality we want, and most importantly, how to use our AI Builder model inside of our app. Let’s get started!

 

Part 3: Creating your Closet Organization Power App

 

How do I create a new Power App using my AI Builder model?

Now that you’ve published your AI Builder model, it is available for use in both Power Apps and Power Automate. Under Use your model, select New app in Power Apps.

You’ll be redirected to your new Power Apps Canvas app, with one screen created for you that has an Object Detector component on it. To connect your AI model to this component, choose Select model in the Properties pane on the right-hand side of your screen, and select the model you created as you followed along with the previous blog post.

Congratulations! You just made your first AI-enabled Power App.

 

 

What features should I build in my Power App?

Building a Power App is an amazing experience because you have the freedom to be creative and build an app that precisely fits your needs. For my app, I wanted to create an experience where I could see what I have in my closet, as well as add new items to my closet and use the AI model to categorize the items properly. Here’s the success criteria I defined for myself for the first version of my Power App:

• Upload an image to the AI model
• Add an image to my closet database with the correct tag
• View items in each tag on a separate screen
• View a list of all the categories I have in my closet

After you define your success criteria, think about how many different screens you’ll need to create in your app to meet the requirements. Use the Tree view on the left-hand side of your screen to create and organize all your screens. For my app, I created the following screens:

• StartScreen: from this screen, I want to be able to add a new item, or see my closet.
• NewItem: from this screen, I want to be able to upload an image to my AI model, tag it, add it to my closet, and then view that category of my closet.
• Categories: this screen will show a list of the tags I have in my closet. Clicking on a tag should take me to a gallery of those items.
• ShowCategory: this screen should show me all the items I have in a particular category.

I’ll walk you through how I built each screen and share tips and tricks I learned along the way.

 

Creating your StartScreen

One of the best things about using Power Apps is the ability to customize your app to make it look exactly how you want. Using the Properties panel on the right-hand side of your screen, design an amazing screen that makes you happy to look at each time you open your app. I used Canva to create some digital assets that make my StartScreen stand out.

 

Remember how I wanted to be able to add an item or see my closet from this screen? I accomplished this by adding two buttons, one for each functionality. Each button uses the Navigate function in the OnSelect property to take me to the NewItem and Categories screens, respectively.

 

 

Here’s the tree view of my StartScreen, where you can see each component that I’ve added to this screen:

 

 

Creating your NewItem screen

Once you’ve created a beautiful start screen, you’ll want to create the screen that you navigate to when you select the Add Item button. Again, feel free to use the Properties pane to customize the look and feel of each page. On my NewItem screen, I have a few different components:

 

 

Let’s start with the most complicated part of this page—adding an item to your closet. In order to be able to display the images you upload in a gallery view in other parts of the app, you need to store the images online. I’m going to walk you through how to store your images in the Microsoft Dataverse, so they can be quickly and easily accessible from your Power App.

1. Go back to the Power Apps home page, and in the left-hand side menu, select Dataverse, then Tables.
2. Create a New table called Closet with two columns – Image, of type Image, and TagName, of type Text. Save your new table, and go back to your Power App.
3. Select Data in the left-hand side menu, and add the new Closets data source to your environment.
4. In the OnSelect property of your Add item to closet button, you’ll use the Patch function to add the TagName and the Image collected by your Object Detection component into your new Dataverse table.

A few things to keep in mind:

• Take note of the properties of the Object Detection component. You can find a full list of the properties in the documentation, but the key ones we’ll work with here are TagName, ObjectCount, and OriginalImage.
• The easiest way to add an image to your database is to create a separate Image control on this screen, set the control’s Image property to ObjectDetector.OriginalImage, and then patch the image from the control to your database.
• You’ll need to use the Patch function with a Record. Take a look at the documentation to Understand tables and records in canvas apps.
• You can check if your OnSelect logic works by navigating to your table in Dataverse, and checking if the data from an image you uploaded has populated in the table.

Next, let’s work on the second button on the page. I want to navigate to the correct category based on what the ObjectDetector identifies as the tag for the image we uploaded. I also want the button to only appear after an image has been tagged. We’ll need to use two properties here—OnSelect and Visible. For the OnSelect property, we want to use the Set function to set a global variable for the item type, varItemType, to the TagName given to us by the Object Detector. We also want to use the Navigate function to take us to the ShowCategory screen. For the Visible property, we’ll use a little trick. Create a new (invisible) Label control at the top of the page and populate it with the TagName, then set the Visible property of the Button to only display if that label isn’t blank.

 

Finally, add a simple icon to Navigate to the StartScreen in case a user changes their mind, and wants to see their closet instead.

 

Creating your Categories screen

Power Apps has a built-in Gallery control that we’ll use here to dynamically populate our categories list with data from our Closet table that’s stored in the Dataverse. After you create a new Gallery control, the key property you’ll want to edit to populate your gallery is the Items property. Set the Items property of the Gallery control to be the combined tag names from the Object Detector control that you have in your app.

 

The other property we’ll want to set here is OnSelect, so that when a user selects a category, they’ll be taken to the page that shows all of the items in that category. Use the Navigate function to set the OnSelect property to go to your ShowCategory screen. You’ll also want to set the varItemType variable that you created while working on the last screen to be equal to the TagName that the user selected.

 

Finally, I added another Home icon that navigates back to the StartScreen.

 

For more information about how to work with and customize galleries, check out the Gallery control documentation, and if you need a refresher on all of the properties of an Object Detector control, check out the documentation on how to use the Object Detector component in a Power App.

 

Creating your ShowCategory screen

The last screen we have to create for our app will show us a gallery of images of the items we have in each category, along with how many items are in that category. For this screen, we’ll create another Gallery control, and set the Items property to use the Filter function, and filter our Closets data source to only show items we want to see. Luckily, we already have a variable that is set to those items, varItemType, so we can use that in our Filter function. We’ll use similar logic to set the Label control at the top of our screen to display the name of the tag, and then use the CountRows function to count the number of records of that type in our data source. Finally, we’ll add an Icon to act as the back button, which uses the Navigate function to take us back to our Categories screen.

 

 

Wrapping it all up

Take the time to test out your app—you can select the Play Icon in the upper-right hand corner to preview your app in the browser, and make sure everything works correctly. You can also use the Stethoscope Icon to initialize the App Checker. In the App checker, you can check the status of all your formulas, runtime, performance, and accessibility in your app. This function in Power Apps works like a debugger, so you are able to identify, understand, and fix any bugs you may encounter while working on your app.

 

When your app is working the way you’d like it to, Save and publish your app. You can then try out your app on your phone, and bring it with you while you organize your closet!

 

In part 4, we’ll reflect on what we’ve learned over the course of this series, and brainstorm more ways to customize our AI-enabled Power App. Looking forward to seeing you here next week!