Delta Lake has gained popularity in recent times due to its unique features and advantages over traditional data warehouse and other storage formats. For those already using traditional data storage format or moving to a lakehouse architecture, Delta Lake can offer several compelling benefits that can further enhance the performance and capabilities of their data pipelines. Many Azure services are integrated with Delta Lake, and now you can use Azure Stream Analytics to write in Delta format.
In this blog, we will explain the native support of Delta Lake in Azure Stream Analytics, that can help users take their workload to the next level, providing a seamless and scalable solution for large-scale data processing and storage. It is easy to start, taking only a few clicks to create an end-to-end pipeline, and write to either a new or existing Delta table stored in Azure Data Lake Storage Gen2.
There are two options to write your output to Delta Lake:
Option 1: Capture Event Hub data to ADLS Gen2 in Delta Lake format, using no-code drag and drop experience.
Option 2: Add ADLS Gen2 as an output. Select Delta Lake as the format and define Delta table path. On ASA job portal, you have full flexibility to configure various aspects of the job.
With both options, you can create an Azure Stream Analytics job, writing to Azure Data Lake Storage Gen2 in Delta format. This Stream Analytics job would run continuously, processing data from the inputs and ingesting to the append-only delta table to the storage account, honoring the batching characteristics set. In addition to raw ingestion, ASA can also process data with a rich set of transformations. This continuously updated delta table can then be consumed from other engines that support delta format for further actions.
Next Step
Learn about the experience following the tutorials below:
Write output data to delta table in ADLS Gen2
Capture Event Hubs data in Delta Lake format