%3CLINGO-SUB%20id%3D%22lingo-sub-974679%22%20slang%3D%22en-US%22%3EADF%20Adds%20Avro%20and%20CosmosDB%20Data%20Flow%20Connectors%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-974679%22%20slang%3D%22en-US%22%3E%3CP%3E%3C%2FP%3E%3CDIV%20class%3D%22video-embed-center%20video-embed%22%3E%3CIFRAME%20class%3D%22embedly-embed%22%20src%3D%22https%3A%2F%2Fcdn.embedly.com%2Fwidgets%2Fmedia.html%3Furl%3Dhttps%253A%252F%252Fwww.youtube.com%252Fwatch%253Fv%253Dplp1etT2ftY%26amp%3Bsrc%3Dhttps%253A%252F%252Fwww.youtube.com%252Fembed%252Fplp1etT2ftY%253Ffeature%253Doembed%26amp%3Btype%3Dtext%252Fhtml%26amp%3Bkey%3Db0d40caa4f094c68be7c29880b16f56e%26amp%3Bschema%3Dyoutube%22%20width%3D%22373%22%20height%3D%22210%22%20scrolling%3D%22no%22%20frameborder%3D%220%22%20allow%3D%22autoplay%3B%20fullscreen%22%20allowfullscreen%3D%22true%22%20title%3D%22Video%22%3E%3C%2FIFRAME%3E%3C%2FDIV%3E%3CP%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20506px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F153872i6714ED9551DCCF18%2Fimage-dimensions%2F506x187%3Fv%3D1.0%22%20width%3D%22506%22%20height%3D%22187%22%20alt%3D%22cosmos1.png%22%20title%3D%22cosmos1.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%3ECosmosDB%20and%20Avro%20formats%20are%20now%20available%20natively%20in%20Data%20Factory's%20Data%20Flows.%20Use%20Mapping%20Data%20Flows%20with%20your%20hierarchies%2C%20arrays%2C%20and%20other%20complex%20data%20types%20to%20generate%20and%20execute%20data%20transformation%20at%20scale.%20Full%20transformation%20capabilities%20are%20supported%3A%20aggregations%2C%20pivots%2C%20joins%2C%20calculate%20columns%2C%20etc.%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20748px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F153914iF826737E5577856B%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22cosmos3.png%22%20title%3D%22cosmos3.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EDynamically%20scale%20your%20ETL%20process%20with%20throughput%20read%20and%20write%20options.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20647px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F153915iAD6DB540828628E1%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22cosmos2.png%22%20title%3D%22cosmos2.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%3E%20CosmosDB%20also%20includes%20dynamic%20throughput%20scale-up%20during%20ETL%20processes.%20Avro%2C%20JSON%2C%20and%20CosmosDB%20support%26nbsp%3B%3C%2FSPAN%3Ehierarchies%20and%20complex%20data%20types%20that%20you%20can%20build%20from%20the%20ADF%20Data%20Flow%20browser%20UI.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThen%2C%20you%20can%20mash%20that%20data%20up%20with%20other%20lake%20and%20database%20sources%20in%20Data%20Flows%20in%20ADF%20that%20leverage%20joins%2C%20aggregations%2C%20window%20analytics%2C%20and%20other%20transformations.%20Full%20support%20for%20upserts%20%26amp%3B%20deletes%20in%20CosmosDB%20is%20also%20supported.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-974679%22%20slang%3D%22en-US%22%3E%3CP%3EAzure%20Data%20Factory%20now%20supports%20Avro%20format%20and%20CosmosDB%20natively%20in%20Mapping%20Data%20Flows%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-974679%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Data%20Factory%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EMapping%20Data%20Flows%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1060551%22%20slang%3D%22en-US%22%3ERe%3A%20ADF%20Adds%20Avro%20and%20CosmosDB%20Data%20Flow%20Connectors%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1060551%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20Mark%2C%20can%20you%20please%20provide%20some%20example%20of%20building%20dynamic%20query%20for%20cosmos%20read%20in%20data%20flow%20expression%20builder.%20I%20have%20a%20need%20where%20timestamp%20value%20comes%20from%20run%20time%20and%20need%20to%20build%20cosmos%20query%20to%20filter%20records.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThanks%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ESarju%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1060936%22%20slang%3D%22en-US%22%3ERe%3A%20ADF%20Adds%20Avro%20and%20CosmosDB%20Data%20Flow%20Connectors%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1060936%22%20slang%3D%22en-US%22%3E%3CP%3EYou%20can%20use%20the%20CosmosDB%20SQL%20Query%20interface%20for%20source%20queries%20as%20described%20here%3A%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fcosmos-db%2Fsql-query-getting-started%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fcosmos-db%2Fsql-query-getting-started%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFor%20example%2C%20this%20will%20work%20against%20my%20moviesdemo%20CosmosDB%20source%20dataset%20in%20an%20ADF%20data%20flow%20Source%20transformation%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%3ESELECT%20c.movies.year%0A%20%20%20%20FROM%20moviesdemo%20c%3C%2FPRE%3E%3C%2FLINGO-BODY%3E
Microsoft

 

cosmos1.png

CosmosDB and Avro formats are now available natively in Data Factory's Data Flows. Use Mapping Data Flows with your hierarchies, arrays, and other complex data types to generate and execute data transformation at scale. Full transformation capabilities are supported: aggregations, pivots, joins, calculate columns, etc.

 

cosmos3.png

Dynamically scale your ETL process with throughput read and write options.

 

cosmos2.png

CosmosDB also includes dynamic throughput scale-up during ETL processes. Avro, JSON, and CosmosDB support hierarchies and complex data types that you can build from the ADF Data Flow browser UI.

 

Then, you can mash that data up with other lake and database sources in Data Flows in ADF that leverage joins, aggregations, window analytics, and other transformations. Full support for upserts & deletes in CosmosDB is also supported.

2 Comments
Occasional Visitor

Hi Mark, can you please provide some example of building dynamic query for cosmos read in data flow expression builder. I have a need where timestamp value comes from run time and need to build cosmos query to filter records.

 

 

Thanks,

 

Sarju

Microsoft

You can use the CosmosDB SQL Query interface for source queries as described here: https://docs.microsoft.com/en-us/azure/cosmos-db/sql-query-getting-started.

 

For example, this will work against my moviesdemo CosmosDB source dataset in an ADF data flow Source transformation:

 

SELECT c.movies.year
    FROM moviesdemo c