Pipeline sometimes needs to understand and reads metadata from trigger that invokes it. For instance, with Tumbling Window Trigger run, based upon window start and end time, pipeline will process different data slices or folders. In Azure Data Factory, we use Parameterization and System Variable to pass meta data from trigger to pipeline.
This pattern is especially useful for Tumbling Window Trigger, where trigger provides window start and end time, and Custom Event Trigger, where trigger parse and process values in custom defined data field. Please note that different trigger types offer different metadata, and System Variable will contain most up to date information for each one.
This section shows you how to pass meta data information from trigger to pipeline, within the Azure Data Factory User Interface.
Go to the Authoring Canvas and edit a pipeline
Click on the blank canvas to bring up pipeline settings. Do not select any activity. You may need to pull up the setting panel from the bottom of the canvas, as it may have been collapsed
Select Parameters section and select + New to add parameters
Add triggers to pipeline, by clicking on + Trigger.
Create or attach a trigger to the pipeline, and click OK
In the following page, fill in trigger meta data for each parameter. Use format defined in System Variable to retrieve trigger information. You don't need to fill in the information for all parameters, just the ones that will assume trigger metadata values. For instance, here we assign trigger run start time to parameter_1.
To pass in trigger information to pipeline runs, both the trigger and the pipeline json need to be updated with parameters section.
{
"name": "demo_pipeline",
"properties": {
"activities": [
{
"name": "demo_activity",
"type": "WebActivity",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"url": {
"value": "@pipeline().parameters.parameter_2",
"type": "Expression"
},
"method": "GET"
}
}
],
"parameters": {
"parameter_1": {
"type": "string"
},
"parameter_2": {
"type": "string"
},
"parameter_3": {
"type": "string"
},
"parameter_4": {
"type": "string"
},
"parameter_5": {
"type": "string"
}
},
"annotations": [],
"lastPublishTime": "2021-02-24T03:06:23Z"
},
"type": "Microsoft.DataFactory/factories/pipelines"
}
{
"name": "trigger1",
"properties": {
"annotations": [],
"runtimeState": "Started",
"pipelines": [
{
"pipelineReference": {
"referenceName": "demo_pipeline",
"type": "PipelineReference"
},
"parameters": {
"parameter_1": "@trigger().startTime"
}
}
],
"type": "ScheduleTrigger",
"typeProperties": {
"recurrence": {
"frequency": "Minute",
"interval": 15,
"startTime": "2021-03-03T04:38:00Z",
"timeZone": "UTC"
}
}
}
}
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.