Automatic pause all Synapse Pools and keeping your subscription costs under control

Published Jan 16 2021 11:21 AM 9,920 Views
Microsoft

As Synapse engineer or Synapse Support Engineer you may need to start and test some Pools, and you want this to be the most cost efficient possible. Leaving some Synapse with a lot of DWU left turned on during the weekend because you forget to pause the DW after you shutdown your computers is not a good approach and we can quickly resolve this by using Powershell + Automation accounts.

 

Before we get into the procedure on the automation, just want to show you some details.

 

We currently have two flavors of Synapse

  • Dedicated SQL pools (formerly SQL DW)
    • Old mode where the SQL DW lives in an Azure SQL DB Server that can be shared with regular Azure SQL databases.
    • Internally is a resource of type (Microsoft.SQL)
    • Sample: /subscriptions/xxxxxx/resourceGroups/yyyyyyy/providers/Microsoft.Sql/servers/yyyyyyyy/databases/olddwpool
  • Azure Synapse Analytics - Dedicated SQL pool
    • SQL DW database inside a workspace
    • Internally is a resource of type (Microsoft.Synapse)
    • Sample: /subscriptions/xxxxxx/resourceGroups/yyyyyyy/providers/Microsoft.Synapse/workspaces/yyyyyyyy/sqlPools/dwpool

 

 

This is important to know because we are going to use 2 different scripts to pause the pool

  • Suspend-AzSynapseSqlPool (Az.Synapse)
  • Suspend-AzSqlDatabase (Az.SQL)

 

 

There are 2 versions of the script

  1. Powershell to run from your machine
  2. Using Azure Automation Account

 

1. Powershell to run from your machine

This script you can run from your machine and you can get last version of script at ScriptCollection/Synapse - Pause all DWs.ps1 at master · FonsecaSergio/ScriptCollection · GitHub

 

You are going to need the modules installed

  • Az.Accounts
  • Az.Sql
  • Az.Synapse

 

 

For below script I'm considering that your user is same that you use to admin your Azure Subscription

 

Find a sample below

Context exists
Current credential is sefonsec@microsoft.com
Current subscription is SEFONSEC Microsoft Azure Internal Consumption
---------------------------------------------------------------------------------------------------
Get SQL / Synapse RESOURCES
---------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
Loop through all Synapse Workspaces
---------------------------------------------------------------------------------------------------
Checking Azure Synapse Workspace [xxxxxxxxxxx_synapse] for Synapse SQL Pools
-> Synapse SQL Pool [dwpool] found with status [Online]
-> Pausing Synapse SQL Pool [dwpool]
-> Synapse SQL Pool [dwpool] paused in 0 hours, 2 minutes and 32 seconds. Current status [Paused]
---------------------------------------------------------------------------------------------------
Loop through all SQL Servers (former SQLDW)
---------------------------------------------------------------------------------------------------
Checking SQL Server [xxxxxxxxxxx-eastus] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools
Checking SQL Server [xxxxxxxxxxx-eastus2] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools
Checking SQL Server [xxxxxxxxxxx-northeu] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools
Checking SQL Server [xxxxxxxxxxx-southcentralus] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools
Checking SQL Server [xxxxxxxxxxx-uksouth] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools
Checking SQL Server [xxxxxxxxxxx-ukwest] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools
Checking SQL Server [xxxxxxxxxxx] in Resource Group [CSSAzureDB] for Synapse SQL Pools
-> Synapse SQL Pool [SQLDW] found with status [Paused]
Checking SQL Server [xxxxxxxxxxx-byok] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools
Checking SQL Server [xxxxxxxxxxx-demo] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools
Checking SQL Server [xxxxxxxxxxx_synapse] in Resource Group [synapseworkspace-managedrg-5da694c3-ae72-4f25-9cc6-626adcf858e6] for Synapse SQL Pools
-> This DB is part of Synapse Workspace - Ignore here Should be done above using Az.Synapse Module
Checking SQL Server [xxxxxxxxxxx-westeu] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools
Checking SQL Server [xxxxxxxxxxx-westus] in Resource Group [CSSAzureDB_OtherRegions] for Synapse SQL Pools

 

A sample with error

Checking Azure Synapse Workspace [xxxxxxxxxxx_synapse] for Synapse SQL Pools
Write-Error: -> Checking Synapse SQL Pool [dwpool] found with status [Resuming]

 

 

2. Using Azure Automation Account

Now we want this to be automated, like as a sample to shutdown every day at 11PM and send me alerts if error happens

 

You can find last version at: ScriptCollection/Synapse - Pause all DWs - Automation Acount.ps1 at master · FonsecaSergio/ScriptCol...

 

1 - Lets first create the Automation Account

2021-01-16 17_03_41-Automation - Microsoft Azure and 7 more pages - Work - Microsoft​ Edge.png

2 - Make sure to create a Run As Account

 

2021-01-16 17_04_25-Add Automation Account - Microsoft Azure and 7 more pages - Work - Microsoft​ Ed.png

 

3 - By default it already got the contributor permission in the subscription level. You can change that if needed. Or add the any required permissions.

2021-01-16 17_17_14-SEFONSEC Microsoft Azure Internal Consumption - Microsoft Azure and 7 more pages.png

 

4 - You need to install the Az modules. Just go to modules and look for them in the gallery.

2021-01-16 17_22_53-SynapseAutomation - Microsoft Azure and 7 more pages - Work - Microsoft​ Edge.png

 

5 - Install first the Az.Account because it's a prerequisite for others

2021-01-16 17_23_38-Browse Gallery - Microsoft Azure and 7 more pages - Work - Microsoft​ Edge.png2021-01-16 17_25_27-Az.Accounts - Microsoft Azure and 7 more pages - Work - Microsoft​ Edge.png

 

Wait for it to complete

2021-01-16 17_26_10-SynapseAutomation - Microsoft Azure and 7 more pages - Work - Microsoft​ Edge.png

6 - Do same for 3 modules

  • Az.Accounts
  • Az.Sql
  • Az.Synapse

 

7 - Now go to runbooks and create a new Powershell runbook

2021-01-16 17_19_01-Create a runbook - Microsoft Azure and 7 more pages - Work - Microsoft​ Edge.png

 

8 - Add code from ScriptCollection/Synapse - Pause all DWs - Automation Acount.ps1 at master · FonsecaSergio/ScriptCol...

 

8.1 - Save and Publish. And click on Start to test it

2021-01-16 17_20_04-Edit PowerShell Runbook_ - Microsoft Azure and 7 more pages - Work - Microsoft​ .png

 

2021-01-16 17_20_50-PauseALLDW (SynapseAutomation_PauseALLDW) - Microsoft Azure and 7 more pages - W.png

 

9 - Check output tab

2021-01-16 17_43_05-PauseALLDW 1_16_2021, 5_42 PM - Microsoft Azure and 6 more pages - Work - Micros.png

 

10 - You can now go to schedules and add a new schedule

2021-01-16 17_44_58-PauseALLDW (SynapseAutomation_PauseALLDW) - Microsoft Azure and 6 more pages - W.png

2021-01-16 17_45_54-New Schedule - Microsoft Azure and 6 more pages - Work - Microsoft​ Edge.png

 

11 - You may also want also to be alerted if schedule fail. Just to back to automation account and click on Alerts

 

2021-01-16 17_46_48-SynapseAutomation - Microsoft Azure and 6 more pages - Work - Microsoft​ Edge.png

 

12 - You need to add a condition when alert will fire and what actions will be taken. This action could be email to you or run some process or run another script

2021-01-16 17_47_40-Create alert rule - Microsoft Azure and 6 more pages - Work - Microsoft​ Edge.png

 

13 - First configure the condition. You are going to use the metric Total Jobs (Like total jobs failed)

2021-01-16 17_48_09-Configure signal logic - Microsoft Azure and 6 more pages - Work - Microsoft​ Ed.png

 

14 - Define

 - runbook name

 - status (Add custom status "Failed")

 - Threshold > 0

 - Run every hour

 

2021-01-16 17_49_13-Configure signal logic - Microsoft Azure and 6 more pages - Work - Microsoft​ Ed.png

 

15 - Add now the action group. In this case, send email to me

 

2021-01-16 17_50_59-Email_SMS message_Push_Voice - Microsoft Azure and 6 more pages - Work - Microso.png

2021-01-16 17_50_35-Create action group - Microsoft Azure and 6 more pages - Work - Microsoft​ Edge.png

 

You are now set. So your DW will be stopped at 11PM if some error happen you will be alerted like as a sample when a DW is in a state that could not be paused.

 

Another last tip is to use the Azure Budget control I got from my colleague @Gonçalo Ventura 

 

Create a budget and alert

The budget puts a maximum limit on the cost of the subscription, in case some service is left running the subscription will automatically suspend when the budget is reached.

To avoid reaching the max limit and let the subscription go into suspended mode, it is possible to create an alert when the cost reaches a percentage of the budget.

To configure a budget, go to “Subscriptions" or “Cost Management + Billing”, then click on Budgets and fill in the parameters for your budget:

 

FonsecaSergio_0-1610820243993.png

 

 

Click next and set an alert:

FonsecaSergio_1-1610820244008.png

 

Because you do not have only Synapse in your subscription, you may have VMs, SQL DBs, etc.

 

 

9 Comments
Co-Authors
Version history
Last update:
‎Sep 15 2021 12:13 PM
Updated by: