Access, visualize and analyze digitally modeled environments as they change over time
As the virtual and physical worlds come together, we are seeing the emergence of a new platform layer called the metaverse. You can use Azure Digital Twins to support metaverse experiences by creating digital models of real-world people, places, and things and bringing these models to life with IoT data. Now with the new data history feature for Azure Digital Twins, you can easily historize digital twin changes over time. Use historical twin data to identify trends and perform anomaly detection in modelled environments, or to even replay metaverse experiences.
The data history feature is powered by an integration between Azure Digital Twins and Azure Data Explorer (ADX) that automatically historizes twin updates to ADX. Once twin property updates are historized, you can run joint queries using the Azure Digital Twins plugin for Azure Data Explorer to reason across digital twins, their relationships, and time series data.
This article shows how to set up a working data history connection between Azure Digital Twins and Azure Data Explorer and then perform historical analytics on a simulated dairy operation that contains factories, people, and machines in multiple locations. It uses the Azure CLI to set up and connect the required data history resources, including:
- an Azure Digital Twins instance
- an Event Hubs namespace containing an Event Hub
- an Azure Data Explorer cluster containing a database
Prerequisites
Set up CLI session
To start working with Azure Digital Twins in the Azure CLI, the first thing to do is log in and set the CLI context to your subscription for this session. Run these commands in your CLI window:
az login
az account set --subscription "<your-Azure-subscription-ID>"
If this is the first time you've used this subscription with Azure Digital Twins, run this command to register with the Azure Digital Twins namespace. (If you're not sure, it's ok to run it again even if you've done it sometime in the past.)
az provider register --namespace 'Microsoft.DigitalTwins'
Next, you'll add the Microsoft Azure IoT Extension for Azure CLI, to enable commands for interacting with Azure Digital Twins and other IoT services. Run this command to make sure you have the latest version of the extension:
az extension add --upgrade --name azure-iot
Now you are ready to work with Azure Digital Twins in the Azure CLI. You can verify this by running az dt --help at any time to see a list of the top-level Azure Digital Twins commands that are available.
Set up local variables for CLI session
This article provides CLI commands that you can use to create the data history resources. In order to make it easy to copy and run those commands later, you can set up local variables in your CLI session now, and then refer to those variables later in the CLI commands when creating your resources.
Note: Update the placeholders (identified with <...> brackets) in the commands below and run these commands to create the variables (if you're using Azure Cloud Shell, run them in the Bash environment). These values will be used later when creating the new resources.
# General Setup
location="<your-resource-region>"
resourcegroup="<your-resource-group-name>"
# Azure Digital Twins Setup
dtname="<name-for-your-digital-twins-instance>"
connectionname="<name-for-your-data-history-connection>"
# Event Hub Setup
eventhubnamespace="<name-for-your-event-hub-namespace>"
eventhub="<name-for-your-event-hub>"
# Azure Data Explorer Setup
clustername="<name-for-your-cluster>"
databasename="<name-for-your-database>"
Create an Azure Digital Twins instance with a managed identity
If you already have an Azure Digital Twins instance, ensure that you have enabled a system-managed identity for it. If you don't have an Azure Digital Twins instance, set one up using the instructions in this section.
Use the following command to create a new instance with a system-managed identity. The command uses three local variables ($dtname, $resourcegroup, and $location) that were created earlier in Set up local variables for CLI session.
az dt create -n $dtname -g $resourcegroup -l $location --assign-identity
Next, use the following command to grant yourself the Azure Digital Twins Data Owner role on the instance. The command has one placeholder, <owneruser@microsoft.com>, that you should replace with your own Azure account information, and uses a local variable ($dtname) that was created earlier in Set up local variables for CLI session.
az dt role-assignment create -n $dtname --assignee "<owneruser@microsoft.com>" --role "Azure Digital Twins Data Owner"
Note: It may take up to five minutes for this RBAC change to apply.
Create an Event Hubs namespace and Event Hub
The next step is to create an Event Hubs namespace and an event hub. This hub will receive digital twin property update notifications from the Azure Digital Twins instance and then forward the messages to the target Azure Data Explorer cluster. The Azure Digital Twins instance also needs to be granted the Azure Event Hubs Data Owner role on the event hub resource in order to set up the data history connection later.
Use the following CLI commands to create the required resources. The commands use several local variables ($location, $resourcegroup, $eventhubnamespace, and $eventhub) that were created earlier in Set up local variables for CLI session.
Create an Event Hubs namespace:
az eventhubs namespace create --name $eventhubnamespace --resource-group $resourcegroup -l $location
Create an event hub in your namespace:
az eventhubs eventhub create --name $eventhub --resource-group $resourcegroup --namespace-name $eventhubnamespace
Create a Kusto (Azure Data Explorer) cluster and database
Next, create a Kusto (Azure Data Explorer) cluster and database to receive the data from Azure Digital Twins. The Azure Digital Twins instance also needs to be granted the Contributor role on the cluster or database, and the Admin role on the database, in order to set up the data history connection later.
Use the following CLI commands to create the required resources. The commands use several local variables ($location, $resourcegroup, $clustername, and $databasename) that were created earlier in Set up local variables for CLI session.
Start by adding the Kusto extension to your CLI session, if you don't have it already.
az extension add -n kusto
Next, create the Kusto cluster. The command below requires 5-10 minutes to execute, and will create an E2a v4 cluster in the developer tier. This type of cluster has a single node for the engine and data-management cluster, and is applicable for development and test scenarios. For more information about the tiers in Azure Data Explorer and how to select the right options for your production workload, see Select the correct compute SKU for your Azure Data Explorer cluster and Azure Data Explorer Pricing.
az kusto cluster create --cluster-name $clustername --sku name="Dev(No SLA)_Standard_E2a_v4" tier="Basic" --resource-group $resourcegroup --location $location --type SystemAssigned
Create a database in your new Kusto cluster (using the cluster name from above and in the same location). This database will be used to store contextualized Azure Digital Twins data. The command below creates a database with a soft delete period of 365 days, and a hot cache period of 31 days. For more information about the options available for this command, see az kusto database create.
az kusto database create --cluster-name $clustername --database-name $databasename --resource-group $resourcegroup --read-write-database soft-delete-period=P365D hot-cache-period=P31D location=$location
Set up a data history connection
Use the command below to create a data history connection between the Azure Digital Twins instance, the Event Hub, and the Azure Data Explorer cluster. By default, this command assumes all resources are in the same resource group as the Azure Digital Twins instance. You can also specify resources that are in different resource groups using the parameter options for this command, which can be displayed by running az dt data-history create adx -h. The command uses several local variables ($connectionname, $dtname, $clustername, $databasename, $eventhub, and $eventhubnamespace) that were created earlier in Set up local variables for CLI session.
az dt data-history connection create adx --cn $connectionname --dt-name $dtname --adx-cluster-name $clustername --adx-database-name $databasename --eventhub $eventhub --eventhub-namespace $eventhubnamespace
When executing the above command, you will be given the option of assigning the necessary permissions required for setting up your data history connection on your behalf. These permissions are granted to the managed identity of your Azure Digital Twins instance. The minimum required roles are:
- Azure Event Hubs Data Owner on the event hub
- Contributor scoped at least to the specified database
- Database principal assignment with role Admin (for table creation / management) scoped to the specified database
After setting up the connection, these roles can be reduced to a single Azure Event Hubs Data Sender role, if desired.
Verify with a sample twin graph
Now that your data history connection is set up, you can test it with data from your digital twins. If you already have twins in your Azure Digital Twins instance that are receiving telemetry updates, you can skip this section and visualize the results using your own resources. Otherwise, continue through this section to set up a sample graph containing twins that can receive telemetry updates.
You can set up a sample graph for this scenario using the Azure Digital Twins Data Simulator. The Azure Digital Twins Data Simulator continuously pushes telemetry to several twins in an Azure Digital Twins instance.
Create a sample graph
You can use the Azure Digital Twins Data Simulator to provision a sample twin graph and push telemetry data to it. The twin graph created here models pasteurization processes for a dairy company.
Start by opening the Azure Digital Twins Data Simulator web application in your browser.
Enter the host name of your Azure Digital Twins instance in the Instance URL field. The host name can be found in the portal page for your instance and has a format like <Azure-Digital-Twins-instance-name>.api.<region-code>.digitaltwins.azure.net. Select Generate Environment.
You will see confirmation messages on the screen as models, twins, and relationships are created in your environment. When the simulation is ready, the Start simulation button will become enabled. Select Start simulation to push simulated data to your Azure Digital Twins instance. To continuously update the twins in your Azure Digital Twins instance, keep this browser window in the foreground on your desktop (and complete other browser actions in a separate window). You can use Azure Digital Twins Explorer to visualize the twin graph.
To verify that data is flowing through the data history pipeline, navigate to the Azure portal and open the Event Hubs namespace resource you created. You should see charts showing the flow of messages into and out of the namespace, indicating the flow of incoming messages from Azure Digital Twins and outgoing messages to Azure Data Explorer.
View the historized twin updates in Azure Data Explorer
In this section, you'll view the historized twin updates being stored in Azure Data Explorer.
Start in the Azure portal and navigate to the Azure Data Explorer cluster you created earlier. Choose the Databases pane from the left menu to open the database view. Find the database you created for this article and select the checkbox next to it, then select Query.
Next, expand the cluster and database in the left pane to see the name of the table. You'll use this name to run queries on the table.
Copy the KQL command below. The command will change the ingestion to batched mode and ingest every 10 seconds.
.alter table <table-name> policy ingestionbatching @'{"MaximumBatchingTimeSpan":"00:00:10", "MaximumNumberOfItems": 500, "MaximumRawDataSizeMB": 1024}'
Paste the command into the query window, replacing the <table-name> placeholder with the name of your table. Select the Run button.
Next, add the following command to the query window, and run it to verify that Azure Data Explorer has ingested twin updates into the table. It may take up to 5 minutes for the first batch of ingested data to appear.
<table_name>
| count
You should see in the results that the count of items in the table is something greater than 0. You can also add and run the following command to view 100 records in the table:
<table_name>
| limit 100
Next, run a query based on the data of your twins to see the contextualized time series data. Use the query below to chart the sum of the outflow of all machines that feed SaltMachine_C0. This Kusto query uses the Azure Digital Twins plugin to select the twins of interest, join those twins against the data history time series in Azure Data Explorer, and then chart the results. Make sure to replace the <ADT-instance> placeholder with the URL of your instance, in the format 'https://<instance-host-name>'.
// Sum the"Outflow" from all machines that feed SaltMachine_C0
let downstreamMachine = "SaltMachine_C0";
let sensor = "OutFlow";
let lookbackPeriod = 1h;
let timeBin = 1m;
let chartTitle = strcat("Total ",sensor, " from all Machines that feed into ", downstreamMachine);
let ADTendpoint = <ADT-instance>;
let ADTquery = strcat(
```SELECT upstreamMachine.$dtId as tid FROM DIGITALTWINS
MATCH (upstreamMachine)-[:feeds]->(downstreamMachine)
WHERE downstreamMachine.$dtId = '```, downstreamMachine, "'");
evaluate azure_digital_twins_query_request(ADTendpoint, ADTquery)
| extend tid_string = tostring(tid)
| join kind=inner (<table_name>
| where TimeStamp > ago(lookbackPeriod)
and Key == sensor)
on $left.tid_string == $right.Id
| extend Value_double = todouble(Value)
| summarize avg(Value_double) by bin(TimeStamp, timeBin), tid_string
| summarize sum(avg_Value_double) by TimeStamp
| render timechart with (title=chartTitle)
The result charts the sum of the outflow from machines feeding SaltMachine_C0.
To keep exploring the dairy scenario, view additional sample queries on GitHub that show how to monitor the performance of twins in the dairy operation based on machine type, factory, maintenance technician, and various relationships between these twins. You can also read a follow up blog on how to chart the time series behavior of digital twins in Grafana.
For more information on using the Azure Digital Twins query plugin for Azure Data Explorer, see the documentation and blog. Be sure to also check out ADX analytics capabilities, including aggregations, time series analysis, anomaly detection and forecasting, geospatial, and more.