Use Spark (Scala) to write data from ADLS to Synapse Dedicated Pool

Published 03-15-2021 06:09 AM 1,234 Views
Microsoft

 

In this article, I would be talking about how can we write data from ADLS to Azure Synapse dedicated pool using AAD . We will be looking at direct sample code that can help us achieve that.

 

1. First step would be to import the libraries for Synapse connector. This is an optional statement.

 

  Mukund_Bhashkar_0-1615791313561.png

2. Next step is to initialize variable to create/read data frames

   Mukund_Bhashkar_1-1615791313568.png

Note : Above step can also be written in below format :

 

//val df = spark.read.csv("abfss://synapse@mukund.dfs.core.windows.net/100SalesRecords.csv")

 

3. Next step would be to use the write api in below format :

 

Mukund_Bhashkar_3-1615791313576.png

Execute the cell and you will be able to see the new table with data popped up:

Mukund_Bhashkar_4-1615791484065.png

Observation in Driver log with this exercise:

 

Mukund_Bhashkar_5-1615791509190.png

We find external data source, file format and external table created as well as dropped during this automated process.

 

More information about other options for dedicated pool and server less related read/write API's in SPARK can be found out on this page.

 

1 Comment
Microsoft

Right now, same activity using SQL server Authentication may fail. This is known issue and being worked upon by us.

%3CLINGO-SUB%20id%3D%22lingo-sub-2210382%22%20slang%3D%22en-US%22%3EUse%20Spark%20(Scala)%20to%20write%20data%20from%20ADLS%20to%20Synapse%20Dedicated%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2210382%22%20slang%3D%22en-US%22%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20this%20article%2C%26nbsp%3BI%20would%20be%20talking%20about%20how%20can%20we%20write%20data%20from%20ADLS%20to%20Azure%20Synapse%20dedicated%20pool%20using%20AAD%20.%20We%20will%20be%20looking%20at%20direct%20sample%20code%20that%20can%20help%20us%20achieve%20that.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E1.%20First%20step%20would%20be%20to%20import%20the%20libraries%20for%20Synapse%20connector.%20This%20is%20an%20optional%20statement.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Mukund_Bhashkar_0-1615791313561.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F263683i0F25A97C5D90F5B0%2Fimage-size%2Fmedium%3Fv%3Dv2%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22Mukund_Bhashkar_0-1615791313561.png%22%20alt%3D%22Mukund_Bhashkar_0-1615791313561.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E2.%20Next%20step%20is%20to%20initialize%20variable%20to%20create%2Fread%20data%20frames%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Mukund_Bhashkar_1-1615791313568.png%22%20style%3D%22width%3A%20701px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F263682i84ECD175BE361E74%2Fimage-dimensions%2F701x25%3Fv%3Dv2%22%20width%3D%22701%22%20height%3D%2225%22%20role%3D%22button%22%20title%3D%22Mukund_Bhashkar_1-1615791313568.png%22%20alt%3D%22Mukund_Bhashkar_1-1615791313568.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3ENote%20%3A%20Above%20step%20can%20also%20be%20written%20in%20below%20format%20%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CDIV%3E%0A%3CDIV%3E%3CSPAN%3E%2F%2Fval%26nbsp%3Bdf%26nbsp%3B%3D%26nbsp%3Bspark.read.csv(%22abfss%3A%2F%2Fsynapse%40mukund.dfs.core.windows.net%2F100SalesRecords.csv%22)%3C%2FSPAN%3E%3C%2FDIV%3E%0A%3C%2FDIV%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E3.%20Next%20step%20would%20be%20to%20use%20the%20write%20api%20in%20below%20format%20%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Mukund_Bhashkar_3-1615791313576.png%22%20style%3D%22width%3A%20449px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F263685i830ED7D43F91A714%2Fimage-dimensions%2F449x55%3Fv%3Dv2%22%20width%3D%22449%22%20height%3D%2255%22%20role%3D%22button%22%20title%3D%22Mukund_Bhashkar_3-1615791313576.png%22%20alt%3D%22Mukund_Bhashkar_3-1615791313576.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EExecute%20the%20cell%20and%20you%20will%20be%20able%20to%20see%20the%20new%20table%20with%20data%20popped%20up%3A%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Mukund_Bhashkar_4-1615791484065.png%22%20style%3D%22width%3A%20580px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F263687i56319F1251265F48%2Fimage-dimensions%2F580x187%3Fv%3Dv2%22%20width%3D%22580%22%20height%3D%22187%22%20role%3D%22button%22%20title%3D%22Mukund_Bhashkar_4-1615791484065.png%22%20alt%3D%22Mukund_Bhashkar_4-1615791484065.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EObservation%20in%20Driver%20log%20with%20this%20exercise%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Mukund_Bhashkar_5-1615791509190.png%22%20style%3D%22width%3A%20715px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F263688i2FF0B97FA69A09C9%2Fimage-dimensions%2F715x143%3Fv%3Dv2%22%20width%3D%22715%22%20height%3D%22143%22%20role%3D%22button%22%20title%3D%22Mukund_Bhashkar_5-1615791509190.png%22%20alt%3D%22Mukund_Bhashkar_5-1615791509190.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EWe%20find%20external%20data%20source%2C%20file%20format%20and%20external%20table%20created%20as%20well%20as%20dropped%20during%20this%20automated%20process.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EMore%20information%20about%20other%20options%20for%20dedicated%20pool%20and%20server%20less%20related%20read%2Fwrite%20API's%20in%20SPARK%20can%20be%20found%20out%20on%20this%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fsynapse-analytics%2Fspark%2Fsynapse-spark-sql-pool-import-export%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3Epage%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2210382%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3ESynapse%20Spark%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ESynapse%20SQL%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Co-Authors
Version history
Last update:
‎Mar 15 2021 12:31 AM
Updated by: