Developer Experiences - Manage Scanning Data Plane using REST APIs.

[Naga_Yenamandra]

We are happy to announce that REST APIs for scanning data plane are now released. Software engineers or developers in your organization can now call these APIs to register data sources, set up scans and classifications programmatically to integrate with other systems or products in your company.

Purview Scaning Data Plane Endpoints

You need to have the purview account name to call scanning APIs. Below is how the endpoint will look:

https://{your-purview-account-name}.scan.purview.azure.com

Set up authentication using service principal.

To call the scanning APIs, the first thing you need to do is to register an application and create a client secret for that application in Azure Active Directory.  When you register an application a service principal is automatically created in your tenant. For more information on how to create a service principal (application) and client secret, please refer here.

Once service principal is created, you need to assign ‘Data source Admin’ role of your purview account to the service principal created above. The below steps need to be followed to assign role to establish trust between the service principal and purview account.

Once service principal is created, you need to assign Data plane roles of your purview account to the service principal created above. The below steps need to be followed to assign role to establish trust between the service principal and purview account.

1. Navigate to your Purview Studio.
2. Select the Data Map in the left menu.
3. Select Collections.
4. Select the root collection in the collections menu. This will be the top collection in the list, and will have the same name as your Purview account.
5. Select the Role assignments tab.
6. Assign the 'Data Source Administrator' role to service principal created above to access various data planes in Purview.

You've now configured the service principal as an application administrator, which enables it to send content to the scanning APIs. Learn about roles here.

Get Token

You can send a POST request to the following URL to get access token.

https://login.microsoftonline.com/{your-tenant-id}/oauth2/token

The following parameters needs to be passed to the above URL.

• client_id:  client id of the application registered in Azure Active directory and is assigned ‘Data Source Admin’ role for the Purview account.
• client_secret: client secret created for the above application.
• grant_type: This should be ‘client_credentials’.
• resource: This should be ‘https://purview.azure.net’

Sample response token:

{

    "token_type": "Bearer",

    "expires_in": "86399",

    "ext_expires_in": "86399",

    "expires_on": "1621038348",

    "not_before": "1620951648",

    "resource": "https://purview.azure.net",

    "access_token": "<<access token>>"

}

Scanning Data Plane REST APIs

Once you have followed all the above steps and have received access token you can now call various scanning APIs programmatically. The different types of entities you can interact with are listed below:

• Classification Rules
• Data Sources
• Key Vault Connections
• Scans and scan related functionality like triggers and scan rule sets.

The below examples explains the APIs you need to call to configure a data source , set up and run a scan for the data source but for complete information on all the REST APIs supported by scanning data plane refer here -

1. To create or update a data source using APIs the following REST API can be leveraged:

PUT {Endpoint}/datasources/{dataSourceName}?api-version=2018-12-01-preview

You can register an Azure storage data source with name ‘myStorage’ by sending a PUT request to the following URL

{Endpoint}/datasources/myStorage?api-version=2018-12-01-preview with the below request body:

{

  "name": "myStorage",

  "kind": "AzureStorage",

  "properties": {

    "endpoint": "https://azurestorage.core.windows.net/"

  }

}

2. To create a scan for a data source already registered in Purview the following REST API can be leveraged:

PUT {Endpoint}/datasources/{dataSourceName}/scans/{scanName}?api-version=2018-12-01-preview

You can schedule a scan ‘myStorageScan’ using a credential ‘CredentialAKV’ and system scan rule set ‘AzureStorage’ for the already registered data source ‘myStorage’ by sending a PUT request to the following URL with the below request body:

{Endpoint}/datasources/myStorage/scans/myStorageScan?api-version=2018-12-01-preview

{

  "kind": "AzureStorageCredential",

  "properties": {

    "credential": {

      "referenceName": "CredentialAKV",

      "credentialType": "AccountKey"

    },

    "connectedVia": null,

    "scanRulesetName": "AzureStorage",

    "scanRulesetType": "System"

  }

}

The above call with return the following response:

{

  "name": "myStorageScan",

  "id": "datasources/myDataSource/scans/myScanName",

  "kind": "AzureStorageCredential",

  "properties": {

    "credential": {

      "referenceName": "CredentialAKV",

      "credentialType": "AccountKey"

    },

    "connectedVia": null,

    "scanRulesetName": "AzureStorage",

    "scanRulesetType": "System",

    "workers": null

  },

  "scanResults": null

}

3. Once the scan is created you need to add filters to the scan which is basically scoping your scan or determining what objects should be included as part of scan. To create a filter, you can leverage the following REST API

PUT {Endpoint}/datasources/{dataSourceName}/scans/{scanName}/filters/custom?api-version=2018-12-01-preview

You can create a filter for the above scan ‘myStorageScan’ by sending a PUT request to the following URL with the below request body. This will create a scope to include folders /share1/user and /share1/aggregated and exclude folder /share1/user/temp/ as part of the scan.

{Endpoint}/datasources/myStorage/scans/myStorageScan/filters/custom?api-version=2018-12-01-preview

{

  "properties": {

    "includeUriPrefixes": [

      "https://myStorage.file.core.windows.net/share1/user",

      "https://myStorage.file.core.windows.net/share1/aggregated"

    ],

    "excludeUriPrefixes": [

      "https://myStorage.file.core.windows.net/share1/user/temp"

    ]

  }

}

The above call will return the following response:

{

  "name": "custom",

  "id": "datasources/myStorage/scans/myStorageScan/filters/custom",

  "properties": {

    "includeUriPrefixes": [

      "https://myStorage.file.core.windows.net/share1/user",

      "https://myStorage.file.core.windows.net/share1/aggregated"

    ],

    "excludeUriPrefixes": [

      "https://myStorage.file.core.windows.net/share1/user/temp"

    ]

  }

}

4.To run a scan, you need to use the following REST API

PUT {Endpoint}/datasources/{dataSourceName}/scans/{scanName}/runs/{runId}?api-version=2018-12-01-preview

You can now trigger the above scan ‘myStorageScan’ by sending a PUT request to the below URL. The runId is a guid.

{Endpoint}/datasources/myStorage/scans/myStorageScan/runs/138301e4-f4f9-4ab5-b734-bac446b236e7?api-version=2018-12-01-preview

The above call will return the following response:

{

  "scanResultId": "138301e4-f4f9-4ab5-b734-bac446b236e7",

  "startTime": "2019-05-16T17:01:37.3089193Z",

  "endTime": null,

  "status": "Accepted",

  "error": null

}

To learn more about Azure Purview, check out our full documentation today.