Forum Discussion

FrancisRomstad's avatar
FrancisRomstad
Copper Contributor
Jan 25, 2022
Solved

Newbe, exporting full result from Azure Synapse Notebook cell

Hi, 

 

I have had success querying data from Customer Insights in Azure Synapse. I have a notebook attached to a sparkpool with language Spark Sql.

Now I have come to a point where I want to get the full result of the query in a cell exported as a csv either in a blob or ftp (or anything).

 

Seems that the result "preview" in the cells is a "TOP 1000". Which will be too few rows.

 

Regards

  • FrancisRomstad's avatar
    FrancisRomstad
    Jan 27, 2022

    Solved it by using PySpark

     

    Added a linked Azure Blob Storage and used this with pandas

     

    %%pyspark

import pandas
df = spark.sql("SELECT string(birthdate) FROM CIDataBase.Customer")
data = df.toPandas()
data.to_csv('abfss://synapseexport/test.csv', storage_options = {'linked_service' :    'AzureDataLakeStorage1Output'})

     

1 Reply

Sort By
  • FrancisRomstad's avatar
    FrancisRomstad
    Copper Contributor
    Jan 27, 2022

    Solved it by using PySpark

     

    Added a linked Azure Blob Storage and used this with pandas

     

    %%pyspark

import pandas
df = spark.sql("SELECT string(birthdate) FROM CIDataBase.Customer")
data = df.toPandas()
data.to_csv('abfss://synapseexport/test.csv', storage_options = {'linked_service' :    'AzureDataLakeStorage1Output'})

     

Resources