Forum Discussion
Data flow sink to Blob storage not writing to subfolder
Hi Everybody
This seems like it should be straightforward, but it just doesn't seem to work... I have a file containing JSON data, one document per line, with many different types of data. Each type is identified by a field named "OBJ", which tells me what kind of data it contains. I want to split this file into separate files in Blob storage for each object type prior to doing some downstream processing. So, I have a very simple data flow - a source which loads the whole file, and a sink which writes the data back to separate files. In the sink settings, I've set the "File name option" setting to "Name file as column data" and selected my OBJ column for the "Column Data", and this basically works - it writes out a separate file for each OBJ value, containing the right data. So far, so good.
However, what doesn't seem to work is the very simplest thing - I want to write the output files to a folder in my Blob storage container, but the sink seems to completely ignore the "Folder path" setting and just writes them into the root of the container. I can write my output files to a different container, but not to a subfolder inside the same container. It even creates the folder if it's not there already, but doesn't use it.
Am I missing something obvious, or does the "Folder path" setting just not work when naming files from column data? Is there a way around this?
1 Reply
This is expected with Name file as column data. In a mapping data-flow sink, the selected column is treated as the complete output path relative to the container, so it overrides the folder path configured on the dataset or sink. That also explains why the folder can be created but remains empty.
The simplest workaround is to build the folder into the filename column. Add a Derived Column such as outputPath:
concat('split/', toString(OBJ), '.json')
Then choose outputPath for Name file as column data. Do not start the value with a slash. You can include additional levels in the same expression if required.
If you actually want one folder per OBJ and Spark-generated filenames inside each folder, use Name folder as column data instead. For your stated requirement—one named file per OBJ—the derived full relative path is the better fit. Also watch for skew: this option reshuffles rows by filename value, so one very large OBJ group can become a performance bottleneck.
