We're excited to announce that continuous export to Delta table is now available in Preview.
Continuous export in ADX allows you to export data from Kusto to an external table with a periodically run query. The results are stored in the external table, which defines the destination, such as Azure Blob Storage, and the schema of the exported data. This process guarantees that all records are exported "exactly once", with some exceptions. Continous export previously supported CSV, TSV, JSON and Parquet formats.
Starting today, you can continuously export to a delta table.
To define continuous export to a delta table:
-
Create an external delta table, as described in Create and alter delta external tables on Azure Storage.
(.create | .alter | .create-or-alter) external table TableName [(Schema)] kind = delta (StorageConnectionString ) [with (Property [, ...])]
-
Define continuous export to this table using the commands described in Create or alter continuous export.
.create-or-alter continuous-export continuousExportName [over (T1, T2 )] to table externalTableName [with (propertyName = propertyValue [, ...])] <| query
Few things to note:
- If the schema of delta table while defining the external table isn’t provided, Kusto will try to infer it automatically based on the delta table defined in the target storage container. The partitioning information will be inferred from the underlying delta table.
- If the schema of delta table while defining the external table is provided and there is no delta table defined in the target storage container, continuous export will create a delta table during the first export.
- The schema of the delta table must be in sync with the continuous export query. If the underlying delta table changes, the export might start failing with unexpected behavior.
Read more : Continuous data export - Azure Data Explorer & Real-Time Analytics | Microsoft Learn
As always, we'd love to hear your feedback and comments.
