SOLVED

Newbe, exporting full result from Azure Synapse Notebook cell

Copper Contributor

Hi, 

 

I have had success querying data from Customer Insights in Azure Synapse. I have a notebook attached to a sparkpool with language Spark Sql.

Now I have come to a point where I want to get the full result of the query in a cell exported as a csv either in a blob or ftp (or anything).

 

Seems that the result "preview" in the cells is a "TOP 1000". Which will be too few rows.

 

Regards

1 Reply
best response confirmed by FrancisRomstad (Copper Contributor)
Solution

Solved it by using PySpark

 

Added a linked Azure Blob Storage and used this with pandas

 

%%pyspark

import pandas
df = spark.sql("SELECT string(birthdate) FROM CIDataBase.Customer")
data = df.toPandas()
data.to_csv('abfss://synapseexport/test.csv', storage_options = {'linked_service' :    'AzureDataLakeStorage1Output'})

 

1 best response

Accepted Solutions
best response confirmed by FrancisRomstad (Copper Contributor)
Solution

Solved it by using PySpark

 

Added a linked Azure Blob Storage and used this with pandas

 

%%pyspark

import pandas
df = spark.sql("SELECT string(birthdate) FROM CIDataBase.Customer")
data = df.toPandas()
data.to_csv('abfss://synapseexport/test.csv', storage_options = {'linked_service' :    'AzureDataLakeStorage1Output'})

 

View solution in original post