Microsoft

First published on MSDN on Jan 15, 2018
To provide complete solution to maintain you Azure SQL DB statistics and maintenance we provide our maintenance script here


In this article, we will explain step by step how to automate this maintenance on Azure (You can also use that to automate your own T-SQL tasks)


General steps:


    1. Create Azure automation account

 

    1. Import SQLServer module

 

    1. Add Credentials to access SQL DB

 

    1. Add a runbook to run the maintenance

 

    1. Schedule task



Step by step instructions:


    1. Create new automation account

 

    1. Login to your Azure portal and click "New" (the green plus sign)

 


    1. Type "automation" in the search box, and choose automation.



      Figure 1 – new automation account

 

    1. Click "create"

 


    1. Fill the form, choose a name for your automation account, and choose in which resource group it will be placed.

      make sure you choose "YES" for the Create Azure Run As account.




      Figure 2 – add automation account form.

 


    1. Click "create" and wait for the account to be created. The new automation configuration blade will be opened once the provision completed.

 

    1. Import SQLServer module

 


    1. Click on "Modules" at the left options panel, and then click on "Browse Gallery" and search for "SQLServer"



      Figure 3 – add module

 


    1. Choose "SqlServer" by matteot_msft



      Figure 4 – module name

 

    1. Then click on "import" and the "OK"

 


    1. Wait for the import to complete

 

    1. Add Credentials to access SQL DB

 


    1. This will use secure way to hold login name and password that will be used to access Azure SQL DB

      You can skip this and use it as clear text if you like to use clear text skip to the next step.

 


    1. Under "Shared Resources" click on credentials



      Figure 5 – Add new credential object

 

    1. Then click on "Add Credential"

 

    1. Type "SQLLogin" as the name of the credential.

 

    1. In the username field type the SQL Login that will be used for maintenance and its password.

 


    1. Click "Create"

 

    1. Add a runbook to run the maintenance

 


    1. Click on "runbooks" at the left panel and then click "add a runbook"



      Figure 6 – Add a runbook

 


    1. Choose "create a new runbook" and then give it a name and choose "Powershell" as the type of the runbook and then click on "create"



      Figure 7 – add new PowerShell runbook

 


    1. Copy and paste the following row to the new runbook.

      Make sure you change your database properties.

       


      $AzureSQLServerName = "<ServerName>" 
      $AzureSQLDatabaseName = "<DatabaseName>" 
      
      $AzureSQLServerName = $AzureSQLServerName + ".database.windows.net" 
      $Cred = Get-AutomationPSCredential -Name "SQLLogin" 
      $SQLOutput = $(Invoke-Sqlcmd -ServerInstance $AzureSQLServerName -Username $Cred.UserName -Password $Cred.GetNetworkCredential().Password -Database $AzureSQLDatabaseName -Query "exec [dbo].[AzureSQLMaintenance] @Operation='all' ,@LogToTable=1" -QueryTimeout 65535 -ConnectionTimeout 60 -Verbose) 4>&1 
      
      Write-Output $SQLOutput​




      TIP : to redirect the verbose output to the runbook log we use the technique as described here

 

    1. Click on Publish and confirm.

 

    1. Schedule task

 


    1. Click on Schedules



      Figure 8 – Schedules

 

    1. Click on "Add a schedule" and follow the instructions to choose existing schedule or create a new schedule.

 


    1. Choose a time when the application is in the idlest figure, as running the maintenance might impact on performance while it's executing.



      Figure 9 – Create new Schedule



Monitoring

you can monitor the success of the job by reviewing the Automation overview page



Figure 10 – Job Overview.

then you can click on each category and drill down...



Figure 11 – Job executions

then you can click on a specific execution and get more details about it including the output of the script



Figure 12 – Details of job execution and output information.

I hope you enjoy this post, please share any thoughts on a comment here in this post.

Notes:

    • Automation account has a limit of 500 minutes of execution time per subscription per month on its free tier. More information about automation account limits can be found here



More information:

Getting Started with Azure Automation

Create a standalone Azure Automation account

My first PowerShell runbook

20 Comments

Hi Yochanan,

 

I am trying to setup this database maintenance job using Runbooks and I am getting the following errors and I am new to Azure and its not clear why I am getting these two errors.  Our server is set to allow remote connections and the server does exist. Could you share your thoughts? Thanks for the help.

 

Error 1:

AzureSQLServerName : The term 'AzureSQLServerName' is not recognized as the name of a cmdlet, function, script file, or
operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try
again.
At line:1 char:1
+ AzureSQLServerName = "*****.database.window ...
+ ~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (AzureSQLServerName:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException

 

2nd error:

 

Invoke-Sqlcmd : A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 0 - No such host is known.) At line:6 char:16 + ... LOutput = $(Invoke-Sqlcmd -ServerInstance $AzureSQLServerName -Userna ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidOperation: (:) [Invoke-Sqlcmd], SqlException + FullyQualifiedErrorId : SqlExceptionError,Microsoft.SqlServer.Management.PowerShell.GetScriptCommand  

 

Thanks,

Kumar.

Occasional Visitor

You need add modules under automation account. 

 

Is there a way I can exclude a particular table from this maintenance job? There is a large table in our database and collecting stats and rebuilding indexes on them takes forever and the job never completes. 

 

Thanks,

Kumar.

Microsoft

@Kumar_Vivekanandam - at this time we do not have that option for the maintenance, however you can edit the maintenance procedure and exclude the table you like to exclude.

 

Thank you @Yochanan_MSFT . I will review the code to check where the table needs to be excluded.

 

Thanks,

Kumar.

Occasional Visitor

@Yochanan_MSFT  does the script take SAMPLE or FULLSCAN as parameters. I couldn't see in the code, but just double checking if this was intentional or just not amended as such yet  ?

Thanks,

 

Chris

Microsoft

@woodsy1978  - I didnt include this as parameter as I would recommend to use FullScan to get better quality for the statistics. 

however, this is T-SQL code so you can modify it for your needs. 

 

HTH, 

Yochanan

Regular Visitor

i followed the steps and getting error error Couldnt find stored procedure  dbo.AzureSQLMaintenance  . How to fix this error>

Microsoft

Hi @sreedxc 

 

please follow the link at the first line in this article ... it will lead you to the procedure page where you can download the code to create the maintenance procedure.

 

here is the link again for your convenience: 

https://blogs.msdn.microsoft.com/azuresqldbsupport/2016/07/03/how-to-maintain-azure-sql-indexes-and-...

Occasional Visitor

if we want to run the proc on  multiple database in azure sql how to run it one by one on each database ?

Occasional Visitor

 also what DB permission are needed ? will the schema /database suffice or we require sa/admin privilege?

Regular Visitor

Hi Yochanan,

Does the proc update both rowstore and column indexes? when i run the query from microsoft site for column store and rowstore

https://docs.microsoft.com/en-us/sql/relational-databases/indexes/reorganize-and-rebuild-indexes?vie...

after  running your maintenance procedure all the columnstore defrag is cleared. However when i run query for defrag on rowstore index still objects show up. Can you check?

 

SELECT a.object_id, object_name(a.object_id) AS TableName,
    a.index_id, name AS IndedxName, avg_fragmentation_in_percent
FROM sys.dm_db_index_physical_stats
    (DB_ID (N'AdventureWorks2016_EXT')
        , OBJECT_ID(N'HumanResources.Employee')
        , NULL
        , NULL
        , NULL) AS a
INNER JOIN sys.indexes AS b
    ON a.object_id = b.object_id
    AND a.index_id = b.index_id;
GO

 

Microsoft

Hi @sripu 

for SQL DB the scope is always one database. (as you know - cross database queries are not available on Azure SQL DB) 

therefore you should run that on each database individually. 

 

Azure SQL Elastic Jobs do great job about it and it can be used to execute the maintenance on your databases. 

 

Regards, 

Yochanan (MSFT)

 
Regular Visitor

Hi Yochanan 

Could you look at my query "after  running your maintenance procedure all the columnstore defrag is cleared. However when i run query for defrag on rowstore index still objects show up. Can you check?

 

Thanks

Sreedhar

Microsoft

Hi @sreedxc 

I'm not sure I understand the question - can you share some more information about it? 

 

Yochanan.

Visitor

Hello Im testing the runbook, but i get the next message. I have already enable the firewall rules and created the user in the database

 

Invoke-Sqlcmd : Cannot open database "********.database.windows.net" requested by the login. The login 

failed.

Login failed for user '******'.

At line:6 char:16

+ ... LOutput = $(Invoke-Sqlcmd -ServerInstance $AzureSQLServerName -Userna ...

+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : InvalidOperation: (:) [Invoke-Sqlcmd], SqlException

    + FullyQualifiedErrorId : SqlExceptionError,Microsoft.SqlServer.Management.PowerShell.GetScriptCommand

 

Invoke-Sqlcmd : 

At line:6 char:16

+ ... LOutput = $(Invoke-Sqlcmd -ServerInstance $AzureSQLServerName -Userna ...

+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : ParserError: (:) [Invoke-Sqlcmd], ParserException

    + FullyQualifiedErrorId : ExecutionFailureException,Microsoft.SqlServer.Management.PowerShell.GetScriptCommand

 Any ideas

 

Thanks  in advance

Occasional Visitor

Can we perform all steps for Azure Managed Database Instance?

I am trying to perform this on Azure Managed database instance. But getting following error message

 

Invoke-Sqlcmd : A network-related or instance-specific error occurred while establishing a connection to SQL Server. 
The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is 
configured to allow remote connections. (provider: TCP Provider, error: 0 - A connection attempt failed because the 
connected party did not properly respond after a period of time, or established connection failed because connected 
host has failed to respond.)
At line:6 char:16
+ ... LOutput = $(Invoke-Sqlcmd -ServerInstance $AzureSQLServerName -Userna ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [Invoke-Sqlcmd], SqlException
    + FullyQualifiedErrorId : SqlExceptionError,Microsoft.SqlServer.Management.PowerShell.GetScriptCommand
 
Invoke-Sqlcmd : 
At line:6 char:16
+ ... LOutput = $(Invoke-Sqlcmd -ServerInstance $AzureSQLServerName -Userna ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ParserError: (:) [Invoke-Sqlcmd], ParserException
    + FullyQualifiedErrorId : ExecutionFailureException,Microsoft.SqlServer.Management.PowerShell.GetScriptCommand
Microsoft

Hi @aarango235 

the error message is clear. login was failed. 

this mainly happen when wrong login name or password was used. 

try to check that.

if you need more help please open support case and we will be more than happy to help you figure out the reason. 

 

Regards,

Yochanan. 

Microsoft

Hi @VaishnavIndurkar 

Yes, you may use this for Managed Instance as well. 

This error is not related specifically for this script but it indicate that connection failed, it seems like network access failed. 

you may access MI using automation account via public endpoint as internal IP is not accessible via Automation account. 

 

Regards, 

Yochanan.