Processing multiple data files dynamically using azure databricks pyspark notebook

%3CLINGO-SUB%20id%3D%22lingo-sub-1664999%22%20slang%3D%22en-US%22%3EProcessing%20multiple%20data%20files%20dynamically%20using%20azure%20databricks%20pyspark%20notebook%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1664999%22%20slang%3D%22en-US%22%3E%3CP%3E%3CSPAN%3EHi%2C%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3EI%20have%20a%20requirement%20to%20process%20multiple%20data%20files%20having%20different%20no%20of%20columns%20using%20azure%20databricks%20pyspark%20notebook%20and%20want%20to%20use%20a%20single%20notrbook%20file%20to%20dynamically%26nbsp%3B%20%26nbsp%3Bprocess%20those%20data%20files%20having%20different%20nos%20of%20columns.%20i%20dont%20want%20to%20write%20multilple%20notebook%20scripts%20to%20process%20hundreds%20of%20files%20.Please%20let%20me%20if%20anyone%20has%20done%20similar%20exercise%20in%20the%20past%20and%20how%20to%20go%20abt%20it.thanks%20a%20lot%20.Shatamjeev%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E
New Contributor

Hi,

I have a requirement to process multiple data files having different no of columns using azure databricks pyspark notebook and want to use a single notrbook file to dynamically   process those data files having different nos of columns. i dont want to write multilple notebook scripts to process hundreds of files .Please let me if anyone has done similar exercise in the past and how to go abt it.thanks a lot .Shatamjeev

0 Replies