How do I go about solving this?
I have the following directory structure on an Azure container:
-Main_Folder
-2021-01
-file1.parquet
-2021-02
-file2.parquet
-file3.parquetwhere the Data is partitioned by year and month to create subfolders. Within these sub-folders, I have my data files. I want to load into my data flow only the latest files that were added within one day from running my data flow pipeline.
I tried using currentUTC() in End Time and subtracting one day -> AddDays(currentUTC(), -1) in Start Time in the 'Filter by last modified' option provided in source options but it didn't work.
I also tried using currentTimestamp() instead but to no avail.
The subfolders are populated by a pipeline that is triggered daily, and the data is partitioned based on year and month. So It is true that I want to target the last folder that I have, but within that folder, I want to load only the files created from the last pipeline run, especially that these subfolders are created dynamically as we go from month to month and year to year.