ADF Data Flow Updates: CDM and Delta Lake Connectors Generally Available

Published Dec 08 2020 01:19 PM 4,464 Views
Microsoft

ADF Data Flow connectors for Common Data Model (CDM) and Delta Lake are both now generally available (GA).

 

This is the documentation link for CDM to learn more about how to read model.json and manifest style of CDM models into ADF. With data flows, you can build powerful ETL processes using CDM formats and then also generate updated manifest files that point to your new, transformed data using CDM as a sink. ADF can use your CDM entity definitions to build ETL projections for transformation and mapping.

 

Here is the documentation link for Delta Lake, which is a Spark-based data lake format that makes working with data in your lake for analytics solution super-easy. With ADF data flows, you can read from Delta Lake folders, transform data, and even update, upsert, insert, delete, and generate new Delta Lake folders using the Delta Lake sink format. You don't need to bring your own Spark cluster, either. ADF provides the Spark compute to create Delta Lake databases.

 

Both of these Data Lake formats are available now in GA under the data flow "Inline Dataset" feature.

%3CLINGO-SUB%20id%3D%22lingo-sub-1968614%22%20slang%3D%22en-US%22%3EADF%20Data%20Flow%20Updates%3A%20CDM%20and%20Delta%20Lake%20Connectors%20Generally%20Available%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1968614%22%20slang%3D%22en-US%22%3E%3CP%3EADF%20Data%20Flow%20connectors%20for%20Common%20Data%20Model%20(CDM)%20and%20Delta%20Lake%20are%20both%20now%20generally%20available%20(GA).%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fformat-common-data-model%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3EThis%20is%20the%20documentation%20link%20for%20CDM%3C%2FA%3E%20to%20learn%20more%20about%20how%20to%20read%20model.json%20and%20manifest%20style%20of%20CDM%20models%20into%20ADF.%20With%20data%20flows%2C%20you%20can%20build%20powerful%20ETL%20processes%20using%20CDM%20formats%20and%20then%20also%20generate%20updated%20manifest%20files%20that%20point%20to%20your%20new%2C%20transformed%20data%20using%20CDM%20as%20a%20sink.%20ADF%20can%20use%20your%20CDM%20entity%20definitions%20to%20build%20ETL%20projections%20for%20transformation%20and%20mapping.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fformat-delta%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3EHere%20is%20the%20documentation%20link%20for%20Delta%20Lake%3C%2FA%3E%2C%20which%20is%20a%20Spark-based%20data%20lake%20format%20that%20makes%20working%20with%20data%20in%20your%20lake%20for%20analytics%20solution%20super-easy.%20With%20ADF%20data%20flows%2C%20you%20can%20read%20from%20Delta%20Lake%20folders%2C%20transform%20data%2C%20and%20even%20update%2C%20upsert%2C%20insert%2C%20delete%2C%20and%20generate%20new%20Delta%20Lake%20folders%20using%20the%20Delta%20Lake%20sink%20format.%20You%20don't%20need%20to%20bring%20your%20own%20Spark%20cluster%2C%20either.%20ADF%20provides%20the%20Spark%20compute%20to%20create%20Delta%20Lake%20databases.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EBoth%20of%20these%20Data%20Lake%20formats%20are%20available%20now%20in%20GA%20under%20the%20data%20flow%20%22Inline%20Dataset%22%20feature.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1968614%22%20slang%3D%22en-US%22%3E%3CP%3EAzure%20Data%20Factory%20has%20updated%20data%20flow%20connectors%20for%20Delta%20Lake%20and%20Common%20Data%20Model%20(CDM)%20as%20Generally%20Available%20(GA)%20for%20both%20sources%20and%20sinks.%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1968614%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Data%20Factory%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%20ETL%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EMapping%20Data%20Flows%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Version history
Last update:
‎Dec 08 2020 07:36 PM
Updated by: