How to start Synapse Pipeline from Rest API
Published Sep 18 2020 10:17 AM 20K Views
Microsoft

Another day another case.

It took me a while to follow this step by step by using the new Synapse APIs. So this post has the intention to make it easier by example.

 

 

You will need:

  • A Synapse workspace
  • An ADF pipeline that you want to start from Rest API.

 

Doc references:

https://docs.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-rest-api

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/rest-api-walkthrough

https://docs.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline/createpipelinerun

 

That is my pipeline code. It is a simple one it just creates Spark Database using a notebook.

 

 

 

 

 

 

%%spark
spark.sql("CREATE DATABASE IF NOT EXISTS DB_example")

 

 

 

 

 

 

As you can see in figure 1. That is my pipeline:

pipelin_noe.png

Figure 1 Pipeline

 

The name of my pipeline is User_not_test.

I can run successfully this pipeline from Synapse Studio. But I want to run it from the Rest API, actually, that is the post idea.

 

Step by Step

The first step consists in using this documentation to register my pipeline/workspace as an application:

(https://docs.microsoft.com/en-us/azure/azure-monitor/platform/rest-api-walkthrough)

 

 

 

 

 

$subscriptionId = "{azure-subscription-id}"
$resourceGroupName = "{resource-group-name}"

# Authenticate to a specific Azure subscription.
Connect-AzAccount -SubscriptionId $subscriptionId

# Password for the service principal
$pwd = "{service-principal-password}"
$secureStringPassword = ConvertTo-SecureString -String $pwd -AsPlainText -Force

# Create a new Azure AD application
$azureAdApplication = New-AzADApplication `
                        -DisplayName "My Azure Monitor" `
                        -HomePage "https://localhost/azure-monitor" `
                        -IdentifierUris "https://localhost/azure-monitor" `
                        -Password $secureStringPassword

# Create a new service principal associated with the designated application
New-AzADServicePrincipal -ApplicationId $azureAdApplication.ApplicationId

# Assign Reader role to the newly created service principal
New-AzRoleAssignment -RoleDefinitionName Reader `
                          -ServicePrincipalName $azureAdApplication.ApplicationId.Guid

 

 

 

 

 

 

Fill the gasp. For example, suppose my workspace was named as synapseworkspace_Demo. So this part of the script  you should add your workspace name as my example:  https://synapseworkspace_Demo.dev.azuresynapse.net

Take note of the password that you created and defined here$pwd = "{service-principal-password}"

 

 

 

 

 

 

# Create a new Azure AD application
$azureAdApplication = New-AzADApplication `
                        -DisplayName "APP_synapseworkspace" `
                        -HomePage "https://synapseworkspace_Demo.dev.azuresynapse.net" `
                        -IdentifierUris "https://synapseworkspace_Demo.dev.azuresynapse.net" `
                        -Password $secureStringPassword

 

 

 

 

 

 

After that, you will execute some steps to actually invoke the API. The example is described here:

https://docs.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-rest-api

But the Synapse APIs are here:

https://docs.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline/createpipelinerun

 

My script will run with the following:

https://synapseworkspace_Demo.dev.azuresynapse.net/pipelines/User_not_test/createRun?api-version=2018-06-01

Do not forget to add the password that you defined: 

$pwd = "Password that you defined on the previous step"

 

 

 

 

 

 

$pwd = "Password that you defined on the previous step"
$azureAdApplication = Get-AzADApplication -IdentifierUri "https://YourWorkspaceName.dev.azuresynapse.net"

$subscription = Get-AzSubscription -SubscriptionId $subscriptionId

$clientId = $azureAdApplication.ApplicationId.Guid
$tenantId1 = $subscription.TenantId
$authUrl = "https://login.microsoftonline.com/${tenantId1}/oauth2/token"
$cred = New-Object -TypeName Microsoft.IdentityModel.Clients.ActiveDirectory.ClientCredential -ArgumentList ($clientId, $pwd)

$AuthContext = [Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext]$authUrl 
$result = $AuthContext.AcquireTokenAsync("https://dev.azuresynapse.net", $cred).GetAwaiter().GetResult()

# Build an array of HTTP header values
$authHeader = @{
'Content-Type'='application/json'
'Accept'='application/json'
'Authorization'=$result.CreateAuthorizationHeader()
}


$request = "https://YourWorkspaceName.dev.azuresynapse.net/pipelines/YourPipelineName/createRun?api-version=2018-06-01"
$body =  @"
{
    "name": "YourWorkspaceName",
    "location": "Region",
    "properties": {},
    "identity": {
        "type": "SystemAssigned"
    }
}
"@


$response = Invoke-RestMethod -Method POST -Uri $request -Header $authHeader -Body $body
$response | ConvertTo-Json
$runId = $response.runId

 

 

 

 

 

 

If you face this error:

Invoke-RestMethod: {"error":{"code":"Unauthorized","message":"The principal 'some number display' does not have the necessary permissions to perform this operation. "}}

 

Go back to you synapse studio -> open Monitoring -> access control and be sure of 2 things:

1) The user  that will start the rest API needs Workspace admin permission

2)The APP that you register needs workspace admin permissions and to satisfy this requisite: Copy the number displayed on the error and add the permission like figure 2:

service principal.pngpermission.png

Figure 2 Permission

 

Note: Use the number of the principal service id, not the APP name to get this permission done.

 

You could also monitor the pipeline execution adding this piece of code:

 

 

 

 

 

while ($True) {
    $response = Invoke-RestMethod -Method GET -Uri "https://YourWorkspaceName.dev.azuresynapse.net/pipelineruns/${runId}?api-version=2018-06-01" -Header $authHeader
    Write-Host  "Pipeline run status: " $response.Status -foregroundcolor "Yellow"

    if ($response.Status -eq "InProgress") {
        Start-Sleep -Seconds 15
    }
    else {
        $response | ConvertTo-Json
        break
    }
}

 

 

 

 

 

 

Here is the execution - Figure 4:

monitoring.png

Figure 4 Execution

 

 

Thanks, Dan Rosales for help during this process.

 

That is it!

Liliam 

UK Engineer

 

 

3 Comments
Version history
Last update:
‎Sep 21 2020 02:13 AM
Updated by: