Home
%3CLINGO-SUB%20id%3D%22lingo-sub-1189736%22%20slang%3D%22en-US%22%3EETL%20in%20the%20Cloud%20is%20Made%20Easy%20Together%20with%20Azure%20Data%20Factory%20and%20Azure%20Databricks%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1189736%22%20slang%3D%22en-US%22%3E%3CP%3EData%20engineering%20in%20the%20cloud%20has%20emerged%20as%20the%20most%20crucial%20aspect%20of%20every%20successful%20data%20modernization%20project%20in%20recent%20years.%20Without%20accurate%20and%20timely%20data%2C%20business%20decisions%20that%20are%20based%20on%20analytical%20reports%20and%20models%20can%20lead%20to%20bad%20results.%20The%20life%20of%20a%20data%20engineer%20is%20not%20always%20glamorous%2C%20and%20you%20don%E2%80%99t%20always%20receive%20the%20credit%20you%20deserve.%20But%20the%20importance%20of%20the%20data%20engineer%20is%20undeniable.%20Microsoft%20Azure%20Data%20Factory's%20partnership%20with%20Databricks%20provides%20the%20Cloud%20Data%20Engineer's%20toolkit%20that%20will%20make%20your%20life%20easier%20and%20more%20productive.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22etl1.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F172850i648F46333BEC5E2B%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22etl1.png%22%20alt%3D%22etl1.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EThe%20combination%20of%20these%20cloud%20data%20services%20provides%20you%20the%20power%20to%20design%20workflows%20like%20the%20one%20above.%20ADF%20has%20built-in%20facilities%20for%20workflow%20control%2C%20data%20transformation%2C%20pipeline%20scheduling%2C%20data%20integration%2C%20and%20many%20more%20capabilities%20to%20produce%20quality%20data%20at%20cloud%20scale%20and%20cloud%20velocity%20all%20from%20a%20single%20pane%20of%20glass.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20you%20are%20a%20data%20developer%20who%20writes%20and%20debugs%20Spark%20code%20in%20Azure%20Databricks%20Notebooks%2C%20Scala%2C%20Jars%2C%20Python%2C%20SparkSQL%2C%20etc.%20you%20can%20point%20to%20your%20data%20routines%20directly%20from%20an%20ADF%20pipeline%20Databricks%20activity.%20Now%2C%20you%20can%20combine%20that%20logic%20with%20any%20of%20the%20other%20activities%20available%20in%20ADF%20including%20looping%2C%20stored%20procedures%2C%20Azure%20Functions%2C%20REST%20APIs%2C%20and%20many%20other%20activities%20that%20allow%20you%20optimize%20other%20Azure%20services%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22etl2.png%22%20style%3D%22width%3A%20156px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F172851i2193E7F028726FB8%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22etl2.png%22%20alt%3D%22etl2.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EADF%20provides%20hooks%20into%20your%20Azure%20Databricks%20workspaces%20to%20orchestrate%20your%20transformation%20code.%20So%2C%20while%20you%20build-up%20your%20extensive%20library%20of%20data%20transformation%20routines%20either%20as%20code%20in%20Databricks%20Notebooks%2C%20or%20as%20visual%20libraries%20in%20ADF%20Data%20Flows%2C%20you%20can%20now%20combine%20them%20into%20pipelines%20for%20scheduled%20ETL%20pipelines.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20you%20prefer%20the%20more%20visually-oriented%20approach%20to%20data%20transformation%2C%20ADF%20has%20built-in%20data%20flow%20capabilities%20that%20provide%20an%20easy-to-code%20UI%20that%20allows%20you%20to%20construct%20complex%20ETL%20process%20like%20this%20generic%20approach%20to%20a%20slowly%20changing%20dimension%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22etl3.png%22%20style%3D%22width%3A%20168px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F172853iD7EFF3C807E103F0%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22etl3.png%22%20alt%3D%22etl3.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22etl4.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F172854i06DCCE85588B0BB4%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22etl4.png%22%20alt%3D%22etl4.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EUse%20the%20ADF%20visual%20design%20canvas%20to%20construct%20ETL%20pipelines%20in%20minutes%20with%20live%20interactive%20debugging%2C%20source%20control%2C%20CI%2FCD%2C%20and%20monitoring.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWhichever%20paradigm%20you%20prefer%2C%20Azure%20Data%20Factory%20provides%20best-in-class%20tooling%20for%20data%20engineers%20who%20are%20tasked%20with%20solving%20complex%20data%20problems%20at%20scale%20using%20Azure%20Databricks%20for%20data%20processing.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1189736%22%20slang%3D%22en-US%22%3E%3CP%3EAzure%20has%20reinvented%20Cloud%20Data%20Engineering%20by%20combining%20the%20power%20and%20ease%20of%20visual%20data%20pipeline%20orchestration%20and%20transformation%20design%20of%20Data%20Factory%20with%20the%20Apache%20Spark-powered%20platform%20of%20Databricks.%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1189736%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Data%20Factory%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%20ETL%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EMapping%20Data%20Flows%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1225957%22%20slang%3D%22en-US%22%3ERe%3A%20ETL%20in%20the%20Cloud%20is%20Made%20Easy%20Together%20with%20Azure%20Data%20Factory%20and%20Azure%20Databricks%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1225957%22%20slang%3D%22en-US%22%3E%3CP%3EIt's%20a%20nice%20article%20however%20my%20question%20is%20that%20nowadays%20we%20can%20do%20most%20of%20the%20data%20transformation%20via%20ADF.%20And%20in%20ADF%20the%20underlying%20technology%20is%20like%20spark%20as%20like%20Databrick.%20Still%20wondering%20why%20do%20we%20need%20Databrick%20in%20this%20architecture%20at%20all%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1226318%22%20slang%3D%22en-US%22%3ERe%3A%20ETL%20in%20the%20Cloud%20is%20Made%20Easy%20Together%20with%20Azure%20Data%20Factory%20and%20Azure%20Databricks%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1226318%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F407167%22%20target%3D%22_blank%22%3E%40avixorld%3C%2FA%3E%26nbsp%3BI%20guess%20you're%20pointing%20towards%20the%20New%20Azure%20Data%20Flow.%20But%20when%20it%20comes%20to%20more%20customized%20ETL%20or%20let's%20say%20a%20heavy%20Data%20Transformation%20(with%20loads%20of%20customized%20Transformations)%20and%20you%20have%20to%20choose%20a%20framework%20like%20Apache%20Spark%20then%20you're%20left%20with%20two%20options...%20Databricks%20or%20HDInsight...%20In%20which%20Databricks%20is%20much%20more%20flexible%20and%20ready-to-use.%20Please%20correct%20me%20if%20I%20am%20wrong.%3C%2FP%3E%3C%2FLINGO-BODY%3E
Microsoft

Data engineering in the cloud has emerged as the most crucial aspect of every successful data modernization project in recent years. Without accurate and timely data, business decisions that are based on analytical reports and models can lead to bad results. The life of a data engineer is not always glamorous, and you don’t always receive the credit you deserve. But the importance of the data engineer is undeniable. Microsoft Azure Data Factory's partnership with Databricks provides the Cloud Data Engineer's toolkit that will make your life easier and more productive.

 

etl1.png

The combination of these cloud data services provides you the power to design workflows like the one above. ADF has built-in facilities for workflow control, data transformation, pipeline scheduling, data integration, and many more capabilities to produce quality data at cloud scale and cloud velocity all from a single pane of glass.

 

If you are a data developer who writes and debugs Spark code in Azure Databricks Notebooks, Scala, Jars, Python, SparkSQL, etc. you can point to your data routines directly from an ADF pipeline Databricks activity. Now, you can combine that logic with any of the other activities available in ADF including looping, stored procedures, Azure Functions, REST APIs, and many other activities that allow you optimize other Azure services:

 

etl2.png

ADF provides hooks into your Azure Databricks workspaces to orchestrate your transformation code. So, while you build-up your extensive library of data transformation routines either as code in Databricks Notebooks, or as visual libraries in ADF Data Flows, you can now combine them into pipelines for scheduled ETL pipelines.

 

If you prefer the more visually-oriented approach to data transformation, ADF has built-in data flow capabilities that provide an easy-to-code UI that allows you to construct complex ETL process like this generic approach to a slowly changing dimension:

 

etl3.png

etl4.png

 

Use the ADF visual design canvas to construct ETL pipelines in minutes with live interactive debugging, source control, CI/CD, and monitoring.

 

Whichever paradigm you prefer, Azure Data Factory provides best-in-class tooling for data engineers who are tasked with solving complex data problems at scale using Azure Databricks for data processing.

 

2 Comments
Regular Visitor

It's a nice article however my question is that nowadays we can do most of the data transformation via ADF. And in ADF the underlying technology is like spark as like Databrick. Still wondering why do we need Databrick in this architecture at all?

Visitor

@avixorld I guess you're pointing towards the New Azure Data Flow. But when it comes to more customized ETL or let's say a heavy Data Transformation (with loads of customized Transformations) and you have to choose a framework like Apache Spark then you're left with two options... Databricks or HDInsight... In which Databricks is much more flexible and ready-to-use. Please correct me if I am wrong.