The Health Bot Service Scenarios
The Health Bot Service is an AI-powered, extensible, secure, and compliant healthcare experience. Microsoft now has a specific template pre-built for assessment purposes. It takes incoming requests, asks about the patient’s symptoms, and aids in getting people access to trusted and relevant healthcare services and information based on the CDC recommendation.
There is a great quick start which covers the deployment of the bot and scenarios here. We also have a Healthcare Bot Reference Architecture and deployment template here.
But how are people using the bot?
After you have deployed the bot to your website and other channels, you really need to understand what people are doing and how it is potentially helping them. In addition, perhaps you are augmenting the data captured by the bot to include user identification, home facility of the user, etc. Questions you may have include:
- How many unique users completed the assessment scenario per day?
- How many unique users started but did not complete the assessment scenario per day?
- For users that did not complete the assessment, at what prompt did they stop?
- For users that completed the assessment, how many were directed to seek medical attention?
- For users that completed the assessment, how many said they have symptoms?
You may also want to filter or slice the above by any of the custom data elements you are capturing.
Insights architecture
In addition to the components that the bot itself is using, we are going to augment those with the following technologies:
- Azure Application Insights
- Power BI Desktop (free download)
- Azure Storage Blobs
Application Insights and Power BI are at the heart of the solution. Application Insights is an application performance management service that collects and analyzes telemetry and logs from your applications. Your bot natively understands how to save its usage telemetry to an Application Insights instance which you create. By default, Application Insights stores data for 90 days. You can choose to change your retention up to 730 days. Alternatively, you can continuously export your telemetry data to the less expensive Azure storage for as long as you choose.
Power BI Desktop is a free tool that can natively connect to Application Insights to create dashboards and reports for local consumption. If you have access to the Power BI service, these reports can be shared with a wide audience by publishing. We have created a Power BI template that answers the questions posed above. You can use this template as-is or customize it based on your unique requirements.
Insights configuration
You will perform the following steps to configure this solution:
- Create an Application Insights resource in Azure
- Configure your bot to write to Application Insights
- Use Power BI Desktop to read data from Application Insights
- Consume and share insights
- (Optional) Use Azure Storage for long term storage of Application Insights data and consumption by other tools
1. Create an Application Insights resource
You should already have a resource group in Azure from your initial bot deployment. From here, click the “Add” button.
Search for Application Insights and click the “Create” button. Give your new resource a name and select the same region where your App Service is deployed. Click “Review + Create” and then “Create” on the following screen.
After your resource is created, open its blade and copy the Instrumentation Key from the overview pane. You will need this for the next step.
2. Configure your bot to write to Application Insights
Open your Health Bot instance. If you are the owner, you should be able to find your bot here. Otherwise the owner will need to provide you the link.
Once your bot is open, navigate to “Integration” -> “Secrets” on the left menu. Select the appropriate option related to capturing conversation text and note the warning about potentially capturing PHI data by doing so. Finally, paste in the Application Insights Instrumentation Key you gathered in the earlier step. Click “Save”.
Even if you are not ready to report on your bot usage data you should enable Application Insights as soon as possible because you will begin tracking usage at once for reporting later. You cannot report on something you did not capture!
3. Use Power BI Desktop to read data from Application Insights
First, you will need download Power BI Desktop here. Next, you will need to download the pre-built Power BI Desktop template (here) that already knows how to pull data from Application Insights and has a report that answers the questions posed at the beginning of this article.
Double click the template (pbit) file you downloaded and enter data into the prompts as shown.
The Application Insights Application ID can be found in the “API Access” section of your Application Insights resource in the Azure portal.
The Number of Days selector specifies how many days’ worth of data will be read from Application Insights into Power BI Desktop. Click Load.
You will then be prompted to authenticate to Application Insights. Click “Organizational account” on the left and click “Connect”. You will need to have been given access to read from the Application Insights resource in the Azure portal before your login will allow you to extract data.
4. Consume and share insights
After your data has loaded, you should see a report like the one shown below.
Keep in mind that most visuals on a Power BI report are clickable and act as filters. For example, if you click one of the days in the bottom left bar chart, the entire report will be filtered to show data for that day. The chart showing completed vs. not completed can also be clicked to act as a filter. This should show you which messages were last shown to the user before the session was abandoned.
If you save this report it will end up as a pbix file on your local file system. You can refresh the data as often as you want by clicking the refresh button on the “Home” ribbon. If you have access to the Power BI service, you can publish this report for broader access within your organization.
Advanced Data & Reporting Topics
We have found that many customers are tracking custom data in their bots. If you need to get custom data from your bot’s initialization payload, this Power BI report makes it easy. On the “Home” ribbon tab, click the “Transform data” button.
Select the “Conversations” query in the left pane and then note the query steps in the right pane. There is a step that brings in the initialization payload, parses its JSON, and then at once removes the parsed JSON record. If you have some custom data you need to gather from this payload, simply delete the step called “Delete Init Payload JSON” (by clicking the X highlighted below in yellow).
From here you can click the expand columns button in the InitPayload column as seen below. Make sure to click “Load more” so that Power BI can get a larger sampling of fields in this JSON object. Select the values you want to extract, uncheck the “use original column name as prefix” checkbox and click “OK”. You may need to set your new columns to appropriate data types. Finally, click the “close & apply” button in the “Home” ribbon tab.
As a final advanced topic, if you find the data as-is does not represent the insights you need to extract, the Application Insights queries are embedded in the Power BI queries. These can be accessed in the Power BI Power Query/Transform Data screen in the “View” ribbon tab and by clicking the “Advanced Editor” button. More information about the Application Insights query language (also referred to as the Kusto Query Language) can be found here.
5. (Optional) Configure Azure Storage for Application Insights data
If you do not want to leverage Power BI for reporting or if you simply want to capture the Application Insights log data for longer than 90 days (or whatever you changed the retention setting to), you can easily configure continuous export. Data stored in Azure storage is extremely inexpensive and can be saved for as long as is needed. In addition, any tool can be used to extract insights from the data stored there including Data Factory, Databricks, etc.
Wrap up and next steps
There are a couple issues that are not necessarily accounted for yet in this report template. For example, if your bot is deployed with multiple languages, the report might not be optimally setup for reporting on outcomes.
In addition, the report is currently setup to pick out anonymous users by checking for a prefix of “anon-“ on the username passed back from the Application Insights query. If you want to use this logic, you will need to configure your bot to write this prefix. If you have alternative logic to determine if a user is anonymous, you may need to change your bot, the Application Insights query, and/or the Power BI report.
All data is captured in Application Insights in the UTC time zone. The data is pulled into Power BI as UTC as well. You should keep this in mind as you are reviewing reports. This is especially relevant if you publish reports to the Power BI service and share with users across multiple time zones.
Finally, feel free to leverage the GitHub repository where we are hosting the template for any issues, suggestions, or feedback. Pull requests are welcomed.
We hope this resource has proved valuable as you assess your health bot usage. Thank you for reading and let us know how we can help.
Todd Kitta, Chief Technical Architect, Microsoft Technology Center
Bryan Roberts, Microsoft Technical Architect, Azure Application Development