Forum Discussion
Unable to write CSV file to Azure Blob Storage using Pyspark
Hi, I am trying to write CSV file to an Azure Blob Storage using Pyspark andI have installed Pyspark on my VM but I am getting this error. org.apache.hadoop.fs.azure.AzureException: com.micro...
Try:
spark = SparkSession.builder \
.config('spark.master', 'local[*]') \
.config('spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version', '2') \
.config("spark.hadoop.mapreduce.fileoutputcommitter.cleanup.skipped", "true") \
.getOrCreate()
spark = SparkSession.builder \
.config('spark.master', 'local[*]') \
.config('spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version', '2') \
.config("spark.hadoop.mapreduce.fileoutputcommitter.cleanup.skipped", "true") \
.getOrCreate()
Amrinder_Singh
Microsoft
MicrosoftOne thing to check is whether you are using a blob storage account or a ADLS Gen 2 (HNS) account. If you are making use of ADLS Gen2 kindly try connecting with ABFS driver instead of WASBS driver.