Forum Discussion

azurepo7's avatar
azurepo7
Copper Contributor
May 24, 2024

Copy files based on wild card file name; from SFTP to Blob storage - Azure Data Factory

I have a list of files on an SFTP server, and I would like to copy and move some of those to the blob storage container. The filenamess should match the wild card: (SCUST_TXT)[0-9]*(-)[0-9]*(-)[0-9]*(.)(CSV).

I have followed the below steps, the pipeline runs fine, but the files are not moved:

  1. created a new pipeline in Azure data factory
  2. added the 'Copy and Move' activity
  3. created source data set and linked service to connect SFTP
  4. created sink data set and linked service to connect blob storage
  5. in the source, i have chosen the wild card path and used the above regex in the wild card file name.

The pipeline runs successfully but it doesn't move the matching files. In the SFTP server i have 3 csv files, but only two of those match the wild card. I was expecting to see those two in the storage container, but it wasn't the case. I'm afraid the wild card pattern is wrong. Here is the sample file name: SCUST_TXT20240417-032938-891.CSV.

If i change the wildcard file name as: *.CSV, the pipeline moves all the three files to the destination (sink). So i assume it's a problem with the wild card.

No RepliesBe the first to reply

Resources