Author: Abid Nazir Guroo is a Program Manager in Azure Synapse Customer Success Engineering (CSE) team.
Azure Synapse Analytics Representational State Transfer (REST) APIs are secure HTTP service endpoints that support creating and managing Azure Synapse resources using Azure Resource Manager and Azure Synapse web endpoints. This article provides instructions on how to setup and use Synapse REST endpoints and describe the Apache Spark Pool operations supported by REST APIs.
In order to perform any operation using Azure REST APIs you need to authenticate the request using an azure active directory authentication token. This token can be generated by various interactive and non-interactive methods. In this tutorial we will use a azure active directory service principal for generating a token with below steps:
You can use various command line utilities like CURL or Powershell, programming languages or GUI clients to interact with REST API endpoints. In this article we will leverage a widely used GUI application - Postman for its simplicity and ease of setup. Postman can be downloaded with this link.
The following parameters are needed to successfully get an Azure AD access token:
Parameter |
Description |
Tenant ID |
The Azure Active Directory (tenant) ID where the application is registered. |
Client ID |
The Application (client) ID for the application registered in Azure AD. |
Client secret |
The Value of the client secret for the application registered in Azure AD. |
grant_type:client_credentials
client_id:<client_id>
client_secret:<Client_secret>
resource:https://management.azure.com/
{
"token_type": "Bearer",
"expires_in": "",
"ext_expires_in": "",
"expires_on": "",
"not_before": "",
"resource": "https://management.azure.com/",
"access_token": "<access token to be used for invoking REST APIs>"
}
Note: The "resource" parameter value should be "https://management.azure.com/" or ""https://management.core.windows.net/" for control plane operations and "https://{workspace name}.dev.azuresynapse.net" for data plane operations.
Apache Spark Pools in Azure Synapse Analytics supports the following REST API operations that can be invoked using a HTTP endpoint:
Operation |
Description |
Method |
API Docs |
Create Or Update |
Create a new Apache Spark Pool or modify the properties of an existing pool. |
PUT |
|
Delete |
Delete a Apache Spark Pool pool. |
DELETE |
|
Get |
Get the properties of a Apache Spark Pool. |
GET |
|
List By Workspace |
List all provisioned Apache Spark Pools in a workspace. |
GET |
|
Update |
Update the properties of an existing Apache Spark pool. |
PATCH |
Detailed documentation of all the REST API operations supported overall by Azure Synapse Analytics can be found with this link.
The Create Or Update REST API can be used to create a new Apache Spark Pool or change configurations of an existing pool, including the ability to upgrade/downgrade the Spark runtime version of a pool. For example, an existing Apache Spark pool with spark runtime version 3.1 can be upgraded to Spark version 3.2 without the need of deletion of the existing pool.
Example:
{
"location": "West US 2",
"properties": {
"sparkVersion": "3.2",
"nodeCount": 6,
"nodeSize": "Medium",
"nodeSizeFamily": "MemoryOptimized",
"autoScale": {
"enabled": false,
"minNodeCount": 6,
"maxNodeCount": 6
},
"autoPause": {
"enabled": true,
"delayInMinutes": 30
}
}
}
Detailed list of configuration properties supported by Apache Spark Pools are documented in this link.
7. Click the send button. A successful creation or modification REST API operation will return a detailed provisioning JSON response like the below sample:
{
"properties": {
"creationDate": "2022-10-28T21:03:07.2066667Z",
"sparkVersion": "3.2",
"nodeCount": 6,
"nodeSize": "Medium",
"nodeSizeFamily": "MemoryOptimized",
"autoScale": {
"enabled": false,
"minNodeCount": 6,
"maxNodeCount": 6
},
"autoPause": {
"enabled": true,
"delayInMinutes": 30
},
"isComputeIsolationEnabled": false,
"sessionLevelPackagesEnabled": false,
"cacheSize": 50,
"dynamicExecutorAllocation": {
"enabled": false
},
"lastSucceededTimestamp": "2022-10-28T22:42:19.23Z",
"isAutotuneEnabled": false,
"provisioningState": "Provisioning"
},
"id": "/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Synapse/workspaces/{workspaceName}/bigDataPools/{bigDataPoolName}",
"name": "{bigDataPoolName}",
"type": "Microsoft.Synapse/workspaces/bigDataPools",
"location": "westus2"
}
Note: You can include an optional Boolean(true/false) parameter in the request URL to stop all currently running sessions/job on the target Spark pool.
The new request URL with forced termination of sessions: https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/provi...}
Azure Synapse Analytics REST APIs REST APIs can be used for operation of various Synapse services. The REST APIs can be used to upgrade/downgrade Spark runtime version, auto-scale configuration, library management and more.
Our team publishes blog(s) regularly and you can find all these blogs here: https://aka.ms/synapsecseblog
For deeper level understanding of Synapse implementation best practices, please refer our Success By Design (SBD) site: https://aka.ms/Synapse-Success-By-Design
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.