Forum Discussion
Root directory path in SFTP Copy Pipeline
Hi There
Please could you help ?
I have a copy pipeline that uses an SFTP location as a source. The SFTP location has multiple folders however I need to read new daily files with changing filenames from the root.
This pipeline was running perfectly until a few days ago when ADF started trying to read from one of the folders instead of the root, despite the root being configured in the source config. I can see from the error message that the pipeline is failing as a result of pipeline reading from a folder instead of the root.
How do I specify the root in the source config of the copy pipeline ?
Thanks
Dom
2 Replies
To specify the root in the source configuration of a copy pipeline that uses an SFTP location with multiple folders, you need to configure the Source tab in your copy activity. Don you can try the steps, Note: This is personal suggestions.
Data store type: Select External.
Connection: Choose your SFTP connection from the list. If it doesn’t exist, create a new one.
File path type: Choose File path, Wildcard file path, or List of files based on your needs.
File path: Manually enter the root path or use the browse option. Wildcard file path: Use wildcards to filter folders or files. For example, root/folder/* to include all files in the folder.
List of files: Specify the folder path and provide a text file with the list of files to copy.Partition root path: When partition discovery is enabled, specify the absolute root path to read partitioned folders as data columns. If not specified, the default is the path you configured
For example, if your root path is root/folder, you would enter this in the File path field. If using wildcards, ensure the path before the first wildcard is correct.
Patooworld Thanks for your input - it looks like removing the Recursively checkbox has helped solve the issue. The files in the root folder were not present every day and it seems the process failed only on days when the files were missing.
