How do I go about solving this?
Jun 09 2021 09:20 AM
I have the following directory structure on an Azure container:
-Main_Folder -2021-01 -file1.parquet -2021-02 -file2.parquet -file3.parquet
where the Data is partitioned by year and month to create subfolders. Within these sub-folders, I have my data files. I want to load into my data flow only the latest files that were added within one day from running my data flow pipeline.
I tried using currentUTC() in End Time and subtracting one day -> AddDays(currentUTC(), -1) in Start Time in the 'Filter by last modified' option provided in source options but it didn't work.
I also tried using currentTimestamp() instead but to no avail.
The subfolders are populated by a pipeline that is triggered daily, and the data is partitioned based on year and month. So It is true that I want to target the last folder that I have, but within that folder, I want to load only the files created from the last pipeline run, especially that these subfolders are created dynamically as we go from month to month and year to year.
How do I go about solving this?
Jun 15 2021 09:46 AM