Retrieve and Reference Trigger Metadata in Pipeline

Pipeline sometimes needs to understand and reads metadata from trigger that invokes it. For instance, with Tumbling Window Trigger run, based upon window start and end time, pipeline will process different data slices or folders. In Azure Data Factory, we use Parameterization and System Variable to pass meta data from trigger to pipeline.

This pattern is especially useful for Tumbling Window Trigger, where trigger provides window start and end time, and Custom Event Trigger, where trigger parse and process values in custom defined data field. Please note that different trigger types offer different metadata, and System Variable will contain most up to date information for each one.

1. Data Factory UI
   
   

   This section shows you how to pass meta data information from trigger to pipeline, within the Azure Data Factory User Interface.
   
   

   1. Go to the Authoring Canvas and edit a pipeline
      
      

   2. Click on the blank canvas to bring up pipeline settings. Do not select any activity. You may need to pull up the setting panel from the bottom of the canvas, as it may have been collapsed
      
      

   3. Select Parameters section and select + New to add parameters
      
      

   4. Add triggers to pipeline, by clicking on + Trigger.
      
      

   5. Create or attach a trigger to the pipeline, and click OK
      
      

   6. In the following page, fill in trigger meta data for each parameter. Use format defined in System Variable to retrieve trigger information. You don't need to fill in the information for all parameters, just the ones that will assume trigger metadata values. For instance, here we assign trigger run start time to parameter_1.
      
      

       

   7. To use the values in pipeline, utilize parameters @pipeline().parameters.parameterName, not system variable, in pipeline definitions. For instance, in our case, to read trigger start time, we'll reference @pipeline().parameters.parameter_1.
   
   
   
2. JSON Schema

To pass in trigger information to pipeline runs, both the trigger and the pipeline json need to be updated with parameters section.

1. Pipeline Definition
   
   Under properties section, add parameter definitions to parameters section
   
   

   {
       "name": "demo_pipeline",
       "properties": {
           "activities": [
               {
                   "name": "demo_activity",
                   "type": "WebActivity",
                   "dependsOn": [],
                   "policy": {
                       "timeout": "7.00:00:00",
                       "retry": 0,
                       "retryIntervalInSeconds": 30,
                       "secureOutput": false,
                       "secureInput": false
                   },
                   "userProperties": [],
                   "typeProperties": {
                       "url": {
                           "value": "@pipeline().parameters.parameter_2",
                           "type": "Expression"
                       },
                       "method": "GET"
                   }
               }
           ],
           "parameters": {
               "parameter_1": {
                   "type": "string"
               },
               "parameter_2": {
                   "type": "string"
               },
               "parameter_3": {
                   "type": "string"
               },
               "parameter_4": {
                   "type": "string"
               },
               "parameter_5": {
                   "type": "string"
               }
           },
           "annotations": [],
           "lastPublishTime": "2021-02-24T03:06:23Z"
       },
       "type": "Microsoft.DataFactory/factories/pipelines"
   }

2. Trigger Definition
   
   Under pipelines section, assign parameter values in parameters section. You don't need to fill in the information for all parameters, just the ones that will assume trigger metadata values.
   
   Here is a Schedule Trigger that will pass its trigger start time
   

   {
       "name": "trigger1",
       "properties": {
           "annotations": [],
           "runtimeState": "Started",
           "pipelines": [
               {
                   "pipelineReference": {
                       "referenceName": "demo_pipeline",
                       "type": "PipelineReference"
                   },
                   "parameters": {
                       "parameter_1": "@trigger().startTime"
                   }
               }
           ],
           "type": "ScheduleTrigger",
           "typeProperties": {
               "recurrence": {
                   "frequency": "Minute",
                   "interval": 15,
                   "startTime": "2021-03-03T04:38:00Z",
                   "timeZone": "UTC"
               }
           }
       }
   }

   
   
3. Use trigger metadata in pipeline
   
   To use the values in pipeline, utilize parameters @pipeline().parameters.parameterName, not system variable, in pipeline definitions.