Background:
Many times, we have a requirement to delete the older logs from $logs as the Retention days is not supported at the moment for ADLS gen2 storage account
Please refer to article https://docs.microsoft.com/en-us/azure/storage/common/manage-storage-analytics-logs?tabs=azure-porta... for more information
Deletion of logs from the $logs can be achieved from the following options
- Azure Storage explorer
- REST API
- SDK’s
- PowerShell
- Azure Data Factory
This blog demonstrates how we can use ADF for deleting logs from $logs
Step 1:
Create an azure data factory resource from azure portal. If you are new to ADF, please refer this link on how to create one: https://docs.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-portal
Step 2:
Open Azure data factory studio in the overview
Step 3:
On the data factory portal click on the pen item -> + > Create pipeline
Step 4:
Click on the Pipeline that has been created -> Under Activities -> General-> select ‘Delete’ and drag it to the pipeline area.
Step 5:
Under Source -> Select New for Datasource =-> choose Azure Data Lake Storage Gen2
Step 6:
In the New linked service, you can use the name that’s by default or customize as per requirement
If the account selection method is azure subscription you can choose the storage account, else you can enter manually too.
Then click on ‘Create’.
Step 7:
In the file path: enter the container as $logs and folder name(according to your requirement)
Step 8:
Coming back to the pipeline, you can set the below properties as per the image :
Step 9 :
Click on Debug, to run the pipeline:
Note : Can make use of Add Trigger to Trigger the pipeline manually
Also you can make use of Add Trigger to Trigger the pipeline on a schedule