How to process files on blob storage based on the FOLDER name?

[zll_0091]

Hi,

I need your expertise on how I can process the files that are in selected folders only.

I am using a notebook to know the max(key) that was loaded in my table.

What I want to do is to process the files that are in the folder that is greater than this max(key).

Ex.

max(key) = 12343

[#2]

Hi @zll_0091 

I did the following --

1.) Get Metadata activity (Argument = Child Items) to retrieve the folder names.

2.) For Each (Items = @activity('Get Metadata1').output.childItems) to loop through the folder names.

3.) In the For Each, add an If Condition with the following expression:

@greater(int(replace(item().name, 'folder', '')), 12343)

In this case I am hardcoding the 12343 key value, you would change this to max(key) or whatever pipeline expression/variable you are using to obtain this (just make sure it is an integer type). Then in the "True" portion of the If statement, you would have your copy activity that references the item().name in the file path. Let me know if you have any questions.

Regards,

Zack

[#3]

Hi @zack-sponaugle 

Thank you so much for your reply. I was stuck in this scenario and been trying the same approach but always receiving an error on the if condition expression. I used the expression that you've provided and it run successfully.

I appreciate your help 🙂