NaN stands for Not a Number. This scenario was a customer trying to insert a parquet file into SQL, but he was not able to do it.
That's because of the following error: Error converting values NaN or Infinity to type 'FLOAT'. NaN and Infinity are not supported. This error was from SQL DW and I got when I tried to run a Select with openrowset pointing to that file.
I added some customization and also I have some nice discussion with my colleague Diya Mothafar. So my colleague mentioned to do the same using Pandas which also valid. My demo will not use Pandas, but again it also does the job.
So the idea here is convert those NaN values into NULL and after that load into SQL Server, we can do this with spark notebooks.
First open Synapse Studio -> Notebook ->PySpark
Fig 1 PySpark
from pyspark.sql.functions import col, when
data_path = spark.read.load('abfss://firstname.lastname@example.org/test/filetest.snappy.parquet', format='parquet')
#here we handle the NaN values
data_path = data_path.replace(float('nan'), None)
Add a Scala cell into the Notebook to add some magic. Note the table will be created with the JOB you do not need to create in advance.
Fig 2 Add Cell
val scala_df = spark.sqlContext.sql ("select * from pysparkdftemptable")
Your new cell should look like this
Fig 3 Cell
Once the Job is complete. You can check the results by opening SSMS and querying the table.
Note: If instead of handling the NaN columns you want to filter it. You can follow the example bellow:
from pyspark.sql.functions import isnan