%3CLINGO-SUB%20id%3D%22lingo-sub-1623444%22%20slang%3D%22en-US%22%3EADF%20adds%20Snowflake%20support%20in%20Mapping%20Data%20Flow%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1623444%22%20slang%3D%22en-US%22%3E%3CP%3EAzure%20Data%20Factory%20now%20enables%20Snowflake%20connector%20in%20Mapping%20Data%20Flow%20to%20expand%20Snowflake%20data%20integration%20support.%20You%20can%20read%20data%20directly%20from%20Snowflake%20for%20analysis%2C%20or%20write%20transformed%20data%20into%20Snowflake%20for%20seamless%20ETL.%26nbsp%3BFor%20other%20Snowflake%20data%20integration%20support%20in%20ADF%2C%20refer%20to%20the%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fazure-data-factory%2Fadf-now-supports-data-integration-with-snowflake%2Fba-p%2F1447370%22%20target%3D%22_blank%22%3Eearlier%20blog%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFor%20example%2C%20when%20using%20Snowflake%20as%20a%20source%20in%20data%20flows%2C%20you%20are%20able%20to%20pull%20your%20data%20from%20a%20table%20or%20via%20custom%20query%2C%20then%20apply%20data%20transformations%20or%20join%20with%20other%20data.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Linda_Wang_0-1598893725741.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F215620i640527CE37D14226%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20title%3D%22Linda_Wang_0-1598893725741.png%22%20alt%3D%22Linda_Wang_0-1598893725741.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAdditionally%2C%20when%20using%20Snowflake%20as%20a%20sink%2C%20you%20can%20perform%20inserts%2C%20updates%2C%20deletes%2C%20and%20upserts%20so%20as%20to%20publish%20the%20analytics%20result%20set%20into%20the%20warehouse.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Linda_Wang_1-1598893725758.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F215621i8E31BAD504F6A750%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20title%3D%22Linda_Wang_1-1598893725758.png%22%20alt%3D%22Linda_Wang_1-1598893725758.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EYou%20can%20point%20to%20Snowflake%20data%20using%20either%20a%20Snowflake%20dataset%20or%20an%20inline%20dataset.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EHere%20is%20a%20walkthrough%20video.%20And%20you%20can%20learn%20more%26nbsp%3Babout%20Snowflake%20support%20in%20Azure%20Data%20Factory%20from%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fconnector-snowflake%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3ESnowflake%20connector%3C%2FA%3E%26nbsp%3Bdocumentation.%3C%2FP%3E%0A%3CP%3E%3C%2FP%3E%3CDIV%20class%3D%22video-embed-center%20video-embed%22%3E%3CIFRAME%20class%3D%22embedly-embed%22%20src%3D%22https%3A%2F%2Fcdn.embedly.com%2Fwidgets%2Fmedia.html%3Fsrc%3Dhttps%253A%252F%252Fwww.youtube.com%252Fembed%252FWrF8Qmc7MEU%253Ffeature%253Doembed%26amp%3Bdisplay_name%3DYouTube%26amp%3Burl%3Dhttps%253A%252F%252Fwww.youtube.com%252Fwatch%253Fv%253DWrF8Qmc7MEU%26amp%3Bimage%3Dhttps%253A%252F%252Fi.ytimg.com%252Fvi%252FWrF8Qmc7MEU%252Fhqdefault.jpg%26amp%3Bkey%3Dfad07bfa4bd747d3bdea27e17b533c0e%26amp%3Btype%3Dtext%252Fhtml%26amp%3Bschema%3Dyoutube%22%20width%3D%22600%22%20height%3D%22337%22%20scrolling%3D%22no%22%20title%3D%22YouTube%20embed%22%20frameborder%3D%220%22%20allow%3D%22autoplay%3B%20fullscreen%22%20allowfullscreen%3D%22true%22%3E%3C%2FIFRAME%3E%3C%2FDIV%3E%3CP%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1623444%22%20slang%3D%22en-US%22%3E%3CP%20style%3D%22margin%3A%200in%3B%22%3E%3CSPAN%20style%3D%22font-size%3A%2011.0pt%3B%20font-family%3A%20'Calibri'%2Csans-serif%3B%22%3EAzure%20Data%20Factory%20enriches%20Snowflake%20data%20integration%20with%20Mapping%20Data%20Flow%20support.%20You%20can%20now%20use%20Snowflake%20connector%20in%20data%20flow%20source%20and%20sink%20transformations.%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1623444%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Data%20Integration%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EBig%20Data%20Analytics%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EMapping%20Data%20Flows%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1670281%22%20slang%3D%22en-US%22%3ERe%3A%20ADF%20adds%20Snowflake%20support%20in%20Mapping%20Data%20Flow%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1670281%22%20slang%3D%22en-US%22%3E%3CP%3EDoes%20ADF%20push%20these%20operations%20(joins%2C%20updates%2C%20merges%2C%20etc.)%20to%20the%20Snowflake%20engine%20(push-down)%20or%20is%20it%20all%20running%20in%20the%20Azure%20servers%3F%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1674335%22%20slang%3D%22en-US%22%3ERe%3A%20ADF%20adds%20Snowflake%20support%20in%20Mapping%20Data%20Flow%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1674335%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F794654%22%20target%3D%22_blank%22%3E%40njarze%3C%2FA%3E%26nbsp%3Bthe%20transformations%20you%20configure%20in%20the%20middle%20of%20source%20and%20sink%20e.g.%20join%20etc%20happens%20on%20ADF's%20managed%20Azure%20Integration%20Runtime.%20For%20the%20step%20of%20writing%20data%20in%20Snowflake%20sink%20-%20insert%2Fupdate%2Fupsert%20(merge)%2Fdelete%20-%20are%20pushed%20down%20to%20Snowflake%20engine.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1989375%22%20slang%3D%22en-US%22%3ERe%3A%20ADF%20adds%20Snowflake%20support%20in%20Mapping%20Data%20Flow%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1989375%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F414697%22%20target%3D%22_blank%22%3E%40Linda_Wang%3C%2FA%3E%26nbsp%3BThanks%20for%20the%20helpful%20blog.%26nbsp%3B%3CBR%20%2F%3E%3CBR%20%2F%3EFor%20transformations%20in%20the%20middle%20of%20source%20and%20sink%2C%20you%20say%20this%20happens%20on%20the%20ADF%20managed%26nbsp%3B%3CSPAN%3EAzure%20Integration%20Runtime.%20Is%20this%20using%20the%20ADF%20Snowflake%20connector%20documented%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fconnector-snowflake%23mapping-data-flow-properties%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3Ehere%3C%2FA%3E%3F%20Is%20this%20connector%20capable%20of%20pushing%20the%20transformation%20down%20so%20that%20it%20is%20executed%20on%20Snowflake%2C%20rather%20than%20pulling%20out%20the%20data%20and%20executing%20on%20a%20spark%20cluster%3F%3CBR%20%2F%3E%3CBR%20%2F%3EEssentially%2C%20I%20would%20like%20to%20know%3A%20is%20it%20possible%20to%20perform%20transformations%20%3CU%3Einside%3C%2FU%3E%20Snowflake%20using%20the%20Mapping%20Data%20Flow%3F%26nbsp%3B%3CBR%20%2F%3E%3C%2FSPAN%3E%3CSPAN%3EI%20would%20prefer%20not%20to%20have%20to%20create%20Azure%20functions%20to%20execute%20Snowflake%20stored%20procedures%2C%20but%20otherwise%20this%20seems%20like%20the%20best%20alternative.%3CBR%20%2F%3E%3CBR%20%2F%3EThank%20you%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E
Microsoft

Azure Data Factory now enables Snowflake connector in Mapping Data Flow to expand Snowflake data integration support. You can read data directly from Snowflake for analysis, or write transformed data into Snowflake for seamless ETL. For other Snowflake data integration support in ADF, refer to the earlier blog.

 

For example, when using Snowflake as a source in data flows, you are able to pull your data from a table or via custom query, then apply data transformations or join with other data.

Linda_Wang_0-1598893725741.png

 

Additionally, when using Snowflake as a sink, you can perform inserts, updates, deletes, and upserts so as to publish the analytics result set into the warehouse.

Linda_Wang_1-1598893725758.png

 

You can point to Snowflake data using either a Snowflake dataset or an inline dataset.

 

Here is a walkthrough video. And you can learn more about Snowflake support in Azure Data Factory from Snowflake connector documentation.

3 Comments
Visitor

Does ADF push these operations (joins, updates, merges, etc.) to the Snowflake engine (push-down) or is it all running in the Azure servers? 

Microsoft

@njarze the transformations you configure in the middle of source and sink e.g. join etc happens on ADF's managed Azure Integration Runtime. For the step of writing data in Snowflake sink - insert/update/upsert (merge)/delete - are pushed down to Snowflake engine.

Senior Member

@Linda_Wang Thanks for the helpful blog. 

For transformations in the middle of source and sink, you say this happens on the ADF managed Azure Integration Runtime. Is this using the ADF Snowflake connector documented here? Is this connector capable of pushing the transformation down so that it is executed on Snowflake, rather than pulling out the data and executing on a spark cluster?

Essentially, I would like to know: is it possible to perform transformations inside Snowflake using the Mapping Data Flow? 
I would prefer not to have to create Azure functions to execute Snowflake stored procedures, but otherwise this seems like the best alternative.

Thank you