Blog Post

Azure Data Factory Blog
1 MIN READ

ADF adds Snowflake support in Mapping Data Flow

Linda_Wang's avatar
Linda_Wang
Icon for Microsoft rankMicrosoft
Aug 31, 2020

Azure Data Factory now enables Snowflake connector in Mapping Data Flow to expand Snowflake data integration support. You can read data directly from Snowflake for analysis, or write transformed data into Snowflake for seamless ETL. For other Snowflake data integration support in ADF, refer to the earlier blog.

 

For example, when using Snowflake as a source in data flows, you are able to pull your data from a table or via custom query, then apply data transformations or join with other data.

 

Additionally, when using Snowflake as a sink, you can perform inserts, updates, deletes, and upserts so as to publish the analytics result set into the warehouse.

 

You can point to Snowflake data using either a Snowflake dataset or an inline dataset.

 

Here is a walkthrough video. And you can learn more about Snowflake support in Azure Data Factory from Snowflake connector documentation.

Updated Sep 01, 2020
Version 2.0
Azure Data Integration
Big Data Analytics
Mapping Data Flows

5 Comments

Sort By
  • njarze's avatar
    njarze
    Copper Contributor
    Dec 03, 2021

    Cause moving the data (that could be "big") from Snowflake to Data Factory, and from Data Factory to Snowflake again could be a bit heavy to maintain 🙂 

     

    Is there something that we can do about it?

    This is exactly the reason we still don't use ADF for our transformation operations. We currently use it to orchestrate our transformations (written in Snowflake SPs), but even for that we are looking at other tools now. 

     

    Hopefully the Data Factory team could implement the "push down" capability before we move. 

  • mmartella's avatar
    mmartella
    Copper Contributor
    Dec 03, 2021

    Hi Linda_Wang, 

    I am also interested in ğŸ™‚

    Maybe the question has been already well explained but I have not found anything about it.

     

    Does ADF let the transformations inside Snowflake? Is there a component that could be implemented to do it?

    Cause moving the data (that could be "big") from Snowflake to Data Factory, and from Data Factory to Snowflake again could be a bit heavy to maintain 🙂 

     

    Is there something that we can do about it?

  • joe-thomas-iw's avatar
    joe-thomas-iw
    Copper Contributor
    Dec 15, 2020

    Linda_Wang Thanks for the helpful blog. 

    For transformations in the middle of source and sink, you say this happens on the ADF managed Azure Integration Runtime. Is this using the ADF Snowflake connector documented https://docs.microsoft.com/en-us/azure/data-factory/connector-snowflake#mapping-data-flow-properties? Is this connector capable of pushing the transformation down so that it is executed on Snowflake, rather than pulling out the data and executing on a spark cluster?

    Essentially, I would like to know: is it possible to perform transformations inside Snowflake using the Mapping Data Flow? 
    I would prefer not to have to create Azure functions to execute Snowflake stored procedures, but otherwise this seems like the best alternative.

    Thank you

  • Linda_Wang's avatar
    Linda_Wang
    Icon for Microsoft rankMicrosoft
    Sep 16, 2020

    njarze the transformations you configure in the middle of source and sink e.g. join etc happens on ADF's managed Azure Integration Runtime. For the step of writing data in Snowflake sink - insert/update/upsert (merge)/delete - are pushed down to Snowflake engine.

  • njarze's avatar
    njarze
    Copper Contributor
    Sep 15, 2020

    Does ADF push these operations (joins, updates, merges, etc.) to the Snowflake engine (push-down) or is it all running in the Azure servers?