Share IoT and Log data in real time using Azure Data Share and Azure Data Explorer
Published Mar 02 2020 01:32 PM 11K Views
Microsoft

In today's digital world, more and more data is generated from devices and software deployed at home and work. This could be data collected from applications, websites, IoT devices and equipment. Often, this data needs to be shared and enriched with 3rd party data in real-time to derive insights. Traditionally, data is shared by generating a data feed made available via file share, FTP, etc. This process requires organizations to invest in maintaining the data pipelines and introduces latency into the system.

 

Azure Data Explorer is a fully managed data analytics service for real-time analysis on large volumes of data streamed from applications and devices. You can use Azure Data Explorer to collect, store, and analyze diverse data on the fly to quickly identify patterns, anomalies and trends. Azure Data Share enables easy and secure data sharing within or cross organizations. You can share data code-free while maintain full visibility into your data sharing relationships including what data is shared and who it is shared with.

 

With the integration between Azure Data Explorer and Azure Data Share, data can be easily and securely shared with partners, service providers or customers for near-real-time collaboration.

 

Here is an example of how organizations with manufacturing plants are using Azure Data Explorer and Azure Data Share for near-real-time analytics. Data is streamed from IoT Hub into Azure Data Explorer cluster and stored in databases. It is then shared through Azure Data Share to the service provider’s Azure Data Explorer cluster for analysis and dashboarding.

 

manufacturing-plant-sharing.png


How data sharing works

Azure Data Share enables sharing of databases in-place from Azure Data Explorer clusters. Data provider can create a share by specifying the databases they want to share, who to share with and terms of use. The same data can be shared with multiple data consumers. Azure Data Share service sends an email invitation to each data consumer, who can then accept the invitation and specify a target Azure Data Explorer cluster in the same Azure data center to access the data.

 

Azure-data-share.png
After sharing relationship is established, Azure Data Share creates a symbolic link between the provider and consumer's Azure Data Explorer clusters. This enables data consumer to read and query the data, but not write to the shared databases. Access to the data uses compute resources from the consumer's Azure Data Explorer cluster. Data consumer can also configure its own access and caching policies to the shared databases, as well as view logs of its own query. When caching policy is configured, data is cached on the consumer's Azure Data Explorer cluster for improved query performance, but is not copied into the blob storage powering the consumer's Azure Data Explorer cluster.

 

adx-sharing-architecture.png


Data provider can stop sharing by revoking access. This will remove the symbolic link and terminate access to the shared database(s).


Data provider share data

To share data, user needs write permission and permission to add role assignment to the source Azure Data Explorer cluster. These permissions typically exist in the Owner role.

 

The following are steps to share data: 

  1. Create an Azure Data Share resource
  2. Within the Azure Data Share resource, create a share by specifying the following information:
    • Name
    • Share type: In-place
    • Description (optional)
    • Terms of use (optional)
    • Azure Data Explorer cluster or databases - When a cluster is shared, all databases on the cluster including future databases created by the data provider are made available to the data consumer. When database is shared, only the individual database is available to the data consumer.
    • Recipients’ Azure login email address


Data consumer receive data

To receive data, user needs write permission and permission to add role assignment to the target Azure Data Explorer cluster. These permissions typically exist in the Owner role.

 

The following are steps to receive data: 

  1. Receive email invitation and follow the link to log into Azure
  2. Agree to the terms of use and accept invitation into its own Azure Data Share resource
  3. Specify an Azure Data Explorer cluster in the same Azure data center as the source to access data


Data consumer can now go to its Azure Data Explorer cluster to grant user permissions to the share databases and access the data. Data ingested using batch mode into the source Azure Data Explorer cluster will show up on the target cluster within a few seconds to a few minutes. 


Additional resources

 

3 Comments
Copper Contributor

Hi @jiefeng,

Do you know when this feature will be publicly available (GA)?

I am looking into a solution like this, can you think of another way to share this data via Azure?

 

Thanks in advance.

Microsoft

@berrybloom Thank you for your questions. The Azure Data Explorer in-place sharing capability is already publicly available. We are currently working towards GA of this capability and will make announcement when it is ready. To answer your second question, there are different ways you can share data located in Azure Data Explorer. For example, extract data from Azure Data Explorer to a storage account for sharing. With the in-place sharing capability, you can easily share data at its source. It enables data recipient to query the data in its original format in near real time. If you are interested in sharing data from sources other than Azure Data Explorer, please reference the Azure Data Share documentation on supported data stores

Microsoft

@berrybloom I want to let you know that Azure Data Explorer in-place sharing capability is now generally available. Please see the announcement blog for details. 

Version history
Last update:
‎Mar 02 2020 01:32 PM
Updated by: