Jan 25 2022 02:43 PM
Hi,
I have had success querying data from Customer Insights in Azure Synapse. I have a notebook attached to a sparkpool with language Spark Sql.
Now I have come to a point where I want to get the full result of the query in a cell exported as a csv either in a blob or ftp (or anything).
Seems that the result "preview" in the cells is a "TOP 1000". Which will be too few rows.
Regards
Jan 27 2022 12:53 AM
SolutionSolved it by using PySpark
Added a linked Azure Blob Storage and used this with pandas
%%pyspark
import pandas
df = spark.sql("SELECT string(birthdate) FROM CIDataBase.Customer")
data = df.toPandas()
data.to_csv('abfss://synapseexport/test.csv', storage_options = {'linked_service' : 'AzureDataLakeStorage1Output'})
Jan 27 2022 12:53 AM
SolutionSolved it by using PySpark
Added a linked Azure Blob Storage and used this with pandas
%%pyspark
import pandas
df = spark.sql("SELECT string(birthdate) FROM CIDataBase.Customer")
data = df.toPandas()
data.to_csv('abfss://synapseexport/test.csv', storage_options = {'linked_service' : 'AzureDataLakeStorage1Output'})