Service Management Automation (SMA) allows you to manage both Microsoft products and external systems. Whether you write your own PowerShell modules or use existing 3 rd party modules, you can easily incorporate them into your SMA runbooks. In some of our demos at TechEd and //Build (on Azure Automation), we have shown how to monitor your Windows Azure Pack (WAP) and Azure resources and send alerts if there are problems. Of course, the same kind of thing can be done for non-Microsoft products. In this post, you will learn how to write a runbook that connects to and monitors Amazon Web Services (AWS) and VMWare virtual machines.
Since this post is on 3 rd party software, I won’t walk you through how to connect to Azure although you could easily include this in your runbook. We do provide an out of box sample in SMA that you can use to connect to Azure called Sample-Managing-Azure.
Runbook OverviewIf you have been reading our blogs, then you know that creating small, modular runbooks is a best practice that we recommend you follow. This instance is an example of how creating small, reusable runbooks can make automation easier. Here is what the runbook flow looks like:
Using this pattern, you can connect to and monitor AWS or VMWare (or Azure) resources depending on which system(s) you are using for your cloud solutions. Each of these runbooks then relies on the Send-Email runbook. Note that you could easily swap out the Send-Email runbook with another runbook that uses a different alerting technique or service. The AWS and VMware runbooks connect to the respective services, and then check to see if the VMs in the service are running. These runbooks return an array of non-running virtual machines to the Watch-VMRunningState runbook. Next, an email is sent to the on-call alias notifying them that VMs powering one of the company’s vital services are down.
Once you have a connection to either AWS or VMware in place, it would be easy to customize these runbooks to meet your individual needs. For example, you could monitor metrics such as memory, CPU, or network IO and call the alert runbook if you have exceeded any of these metrics. You could also easily add a logging runbook that keeps track of performance information across your VMs. In this way, you would ensure that you have a consistent solution across all of your services, regardless of the cloud platform used.
To follow along with the example in this post, download the runbooks from Script Center here . If you do not use AWS or VMware, you can simply ignore that sample and comment out the code in the Watch-VMRunningState runbook.
Import and publish the runbooks in the following order so that the child runbooks are published before the parent runbook:
- Send-Email
- Get-AwsProblemVMs
- Get-VMwareProblemVMs
- Watch-VMRunningState
Before you publish the “Send-Email” runbook, make sure to edit the $SmtpService email service in the runbook. You also need to create a PS Credential asset named “EmailCredentials” to use to connect with your email service, a string asset named “VMwareOnCallEmail” for the email sent for VMware issues, and a string asset named “AWSOnCallEmail” for the email sent to AWS VM issues.
AWSThis section will provide details on how to get set up in SMA so that you can manage AWS resources. If you are only interested in VMware, skip to the next section.
Setting up the AssetsTo start, you need to have an AWS subscription and you should have created at least one EC2 instance in order to follow along with this example. You also need the AWS PowerShell module.
Follow these steps to download and import the AWS module:
1. Download Developer Tools from http://aws.amazon.com/powershell/
2. Once you download the toolkit, the module is located in the “...\AWS Tools\PowerShell” folder.
3. Zip the AWSPowerShell folder
4. To import the module, go to the assets tab and select “Import”. Select the AWSPowerShell.zip file.
In order to connect to your AWS account from PowerShell, you will need to specify your access key and your secret key. You can follow instructions here on http://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-signup.html to create a new IAM account and/or generate a new access/secret key pair.
Once you know your access keys, go to Assets in SMA. Create a PS Credential with your AWS Access key as the username and the AWS Secret key as the password. I named my credential “AWSCredentials”. If you decide to name your credential differently and are using the samples, make sure to update the PS Credential name in the Get-AwsProblemVMs runbook.
Monitoring AWS ResourcesNow that you have created all the assets needed to communicate with AWS, let’s open the Get-AwsProblemVMs runbook and take a look at what it does.
First, the runbook retrieves the values of the keys that you just stored and sets the value for the default AWS region:
|
#Retrieve the values for the access and secret key
$Creds = Get-AutomationPSCredential -Name 'AWSCredentials' $AccessKey = $Creds.UserName $SecretKey = $Creds.GetNetworkCredential().Password $DefaultRegion = "us-west-1" |
The next portion of code is contained in InlineScript. In this section, the runbook sets the access keys for the session and the default region, so that you do not need to refer to them with every command:
|
#Set the credentials and default region for the session
Set-AWSCredentials -AccessKey $using:AccessKey -SecretKey $using:SecretKey
Set-DefaultAWSRegion -Region $using:DefaultRegion
|
If you are interested in learning more about setting access keys using PowerShell, you can read about it here .
You can then retrieve all of the instances in each of your EC2 Regions and check the state of the machine. Since Instances.State is an object, it is deserialized outside of the InlineScript. Because of this, the runbook only stores the instance ID if the VM is not running and then returns the array of problem instances. $VMsWithProblems is then returned from InlineScript and from the Get-AwsProblemVMs runbook.
|
#Get each instance in every region & check if it is running
$Instances = (Get-EC2Region | Get-EC2Instance).Instances ForEach ($Instance in $Instances) { if ($Instance.State.Name -ne "running") { $VMsWithProblems += $Instance.InstanceID } } |
Finally, in the Watch-VMRunningState runbook, if any of the VMs were identified as not running, the Send-Email runbook is called:
|
if ($checkAWS) {
#List of new failed VMs $InstanceIDs = @() $HasNewFailures = $false #Get the machine IDs of any VMs that are not running $StoppedAWSVMs = Get-AwsProblemVMs ForEach ($VM in $StoppedAWSVMs) { #Remove VMs for which notifications have been sent if ($DownAwsMachines -NotContains $VM) { "AWS VM instance: $VM is not responding" $InstanceIDs += $VM $HasNewFailures = $true } } #Send email alert if ($HasNewFailures) { $AWSOnCallEmail = Get-AutomationVariable -Name 'AWSOnCallEmail' $Message = "AWS Instance ID(s): `r" + ($InstanceIDs -join "`r") $Subject = "Warning - AWS VM is down!" Send-Email -Body $Message -Subject $Subject -SendTo $AWSOnCallEmail } $DownAwsMachines = $StoppedAWSVMs } |
VMware
Now, let’s move on to monitoring VMware resources. I’ll walk you through downloading the PowerCLI snap-in, converting it to a portable module to use in SMA, connecting to your server and checking your VM status.
Setting up the AssetsIn order to use the PowerCLI cmdlets, you must first have the VMWare PowerCLI installed on your computer or on each of the runbook workers that power SMA. You can download PowerCLI here: https://my.vmware.com/web/vmware/details?downloadGroup=PCLI550&productId=353 .
Since PowerCLI is a snap-in and relies on its DLLs to run, you must take an extra step in order to use the cmdlets in SMA – convert PowerCLI into a portable module. Portable modules allow you to import and use modules that would otherwise be difficult to import into SMA. As mentioned above, you can either install PowerCLI on each or your runbook workers or you can install the cmdlet on another computer and use InlineScript to reference that module there. In this example, I remote to my VMWare server. Check out this blog post if you want to learn more about portable modules.
Use the New-SmaPortableModule cmdlet in the SMA PowerShell module to create the portable module that you will load in SMA.
Code snippet:
|
$session = New-PSSession New-SmaPortableModule -Session $session -ModuleName "VMware.VimAutomation.Core" -OutputPath "C:\Users\elcooper\Desktop" -IsSnapIn |
Screenshot:
You should now find the .zip file located in the OutputPath that you specified. Now, you are ready to begin your work in SMA and upload the VMware portable module to the Assets store.
In addition to the VMware portable module, the Get-VMwareProblemVMs runbook also uses two variables for the server name and credentials. I used a string called “VMwareServer” for the server name and a PS Credential named “VMWareCredentials” for my user vSphere username and password. If you are using the sample runbooks and would like to use different variables, make sure to update these names at the beginning of your runbook.
Monitoring VMware ResourcesThe Get-VMwareProblemVMs runbook pulls the server name, user name and password that you stored in Assets and then checks the state of all the VMs on this server. It then returns any machines that are not running. Each of the PowerCLI cmdlets must be in an inlinescript. If you installed PowerCLI on a VM host, you will need to reference this host as follows:
InlineScript {
#Some PowerCLI command goes here
} -PSComputerName "MyHost"
Just like the AWS runbook, this runbook first retrieves the assets containing your subscription information:
|
#Retrieve server and credentials from Assets
$Server = Get-AutomationVariable -Name 'VMwareServer' $Creds = Get-AutomationPSCredential -Name 'VMWareCredentials' $Username = $Creds.UserName $Password = $Creds.GetNetworkCredential().Password |
The majority of the rest of the work is done in the InlineScript portion of the code since the VMware module that you are referencing is a portable module. First, you must add the snap-in to the session if you have not added it to your PowerShell profile on the machine PowerCLI is installed on. After that, you can connect to the server:
|
Add-PSSnapin VMware.VimAutomation.Core
#Connect to the server if not connected. if ($DefaultVIServers.Count -lt 1) { Connect-VIServer -Server $using:Server -User $using:Username -Password $using:Password } |
Once you are connected to the server, you can perform actions against resources on the server. The sample runbook checks each VM on your vSphere host and stores the machine name if it is not running:
|
#Get each VM & make sure it is up and running
$VMs = Get-VM $VMsWithProblems = @() foreach ($VM in $VMs) { if ($VM.powerstate -ne "PoweredOn") { $VMsWithProblems += $VM.Name } } |
The Get-VMwareProblemVMs runbook then returns any of the VMs that were not running to the Watch-VMRunningState runbook. An email is then sent for the results if one has not already been sent:
|
if ($checkVMware) {
#List of new failed VMs $VMs = @() $HasNewFailures = $false #Get the machine names of any VMs that are not running $StoppedVMwareVMs = Get-VMwareProblemVMs ForEach ($VM in $StoppedVMwareVMs) { #Remove VMs for which notifications have been sent if ($DownVMwareMachines -NotContains $VM) { "VMware VM is not responding" $VMs += $VM $HasNewFailures = $true } } #Send email alert if one has not already been sent if ($HasNewFailures) { $VMwareOnCall = Get-AutomationVariable -Name 'VMwareOnCallEmail' $Message = "VMware VM(s): `r" + ($VMs -join "`r") $Subject = "Warning - VMware VM is down!" Send-Email -Body $Message -Subject $Subject -SendTo $VMwareOnCall } $DownVMwareMachines = $StoppedVMwareVMs } |
Conclusion
If you use VMware or AWS you can use these samples as a reference to help you manage these resources from SMA. As you can see from this post, using SMA you can develop consistent solutions across a variety of resources whether they are Microsoft or 3rd party software. What other 3 rd party systems do you use? Feel free to provide feedback so we can continue to provide relevant samples to help make your day easier.