Using Azure Functions to read AVS data via Powershell, PowerCLI, and the API

Gotcha’s

• Functions are region specific.
• Functions require a small (ie /28)
• Do not use “ –  “ if Azure functions, use “ _ “ if you need a spacer.
• Pay attention to the sections on RBAC and system managed identities.

 

Build Order

 

1. Create or identify the Log Analytics Workspace that will be used for storage diagnostics.

 

2. Create a KeyVault and set the RBAC properties to allow creation of secrets.

 

3. Create KeyVault secrets to hold the account IDs and account passwords that will be used to authenticate to the vCenter, NSX-T and HCX appliances.

 

4. Create/identify a storage account and an empty subnet that will be used by the function.

 

5. Create the Function App

 

6. Creating the Functions App common variable retrieval code (shared by all functions)

 

7. Example 1: Retrieving vCenter data

 

8. Example 2: Retrieving NSXT data

 

9. Example 3: Retrieving PowerCLI data from HCX

 

 

 

 

 

 

1.      Create or identify a Log Analytics Workspace

If you are running Azure VMware Solution already, you should have Diagnostics enabled and be forwarding logs to that workspace.

To identify the workspace, select the ‘Diagnostics’ option and record the name of the log space.  In this case it is ‘AVSRelatedLogsOnly’.

 

 

If you have not yet enabled diagnostics, follow the instructions found here.  Configure Azure VMware Solution diagnostics.

 

Do not proceed to the next step until this is complete.

 

 

 

 

2.      Create a KeyVault

Create a KeyVault if there isn’t one already.  This example uses the default settings so once the Basics tab is complete, select the ‘Review + Create’, then ‘Create’.

 

  

 

The summary page should look like this

 

 

 

 

 

Note, the creation of private endpoints for this scenario will be covered in the future.

 

 

 

Before you can store secrets RBAC settings need to be applied because by default the user who creates the KeyVault cannot create secrets.

1. a) Go to Access Control, add a Role Assignment, add yourself as the KeyVault Administrator.

 

 

 

1. b) Then click ‘Review and assign’

 

1. c) Next, switch back to the function, and under Identity, select System assigned, then click Save.

 

 

Note that this will use this identity for every function in the function app

 

 

 

1. d) Once that has been created – click on the ‘Azure role Assignment’

 

 

1. e) Select ‘Add role assignment’ then select a scope of ‘KeyVault’, make sure you’ve got the right subscription, and then select name of the KeyVault you are using. The Role required is ‘Key Vault Secrets User’

 

 

1. f) Click ‘Save’

 

3.      Add KeyVault Secrets

1. a) Return to KeyVault and select ‘Secrets’. Now you should be able to click on the ‘Generate/Import’ option to create a secret. If this fails, go back and check the role assignment requirements above.

 

 

 

 

 

1. b) Using the Generate/Import we will fill the following five secrets, one for each variable required in the function: $sharedKey, $avs-VCenterAdmin, $avs-VCenterAdminPassword, $avs-NSXTAdmin, $avs-NSXTAdminPassword.

 

 

Note:  There a number of options here.  In this example we are using the CloudAdmin@vsphere.local to access the vCenter and HCX APIs.  For the NSX-T, we will use the CloudAdmin account.  In production you would likely use an LDAP account with administrative access to both environments, or two separate accounts for added security.

 

1. $sharedKey

 

Open a second portal tab and go to the log analytics workspace we identified above called AVSRelatedLogsOnly and select Agents, then copy the primary key.

 

 

 

Return to the Generate/Import screen and fill in the Name “sharedKey” and the Secret value with the primary key copied from log analytics workspace.  Fill in the content type description.

 

 

 

Click Create

 

 

1. $avs-vCenterAdmin

 

Click on the Generate/Import to create the next secret.  For queries that will connect to vCenter or HCX go to the AVS SDDC instance in the portal, select VMware Credential, and then copy the username, ‘cloudadmin@vsphere.local’ .  Note that in production you should use an identity that has read-only rights.

 

Add the ‘cloudadmin@vsphere.local” as the Secret value for the avs-vCenterAdmin secret.

 

 

Click Create

 

1. $avs-vCenterAdminPassword

 

Click on the Generate/Import to create the next secret.  For queries that will connect to vCenter or HCX go to the AVS private cloud instance in the portal, select ‘VMware credentials’, and then copy the cloudadmin@vsphere.local password.  Paste it in as the ‘Secret value’

 

 

 

Click Create

 

1. $avs-NSXTAdmin

 

Add the user account that can read the NSXT data. In this example, the NSXT cloudadmin account is being used. 

 

 

Click Create

 

 

1. $avs-NSXTAdminPassword

Add the password for the user account that will be used to read the NSXT data.

 

 

             Click ‘Create’

When this is complete, the Secrets view of the KeyVault should look something like this.

 

 

 

 

4.      Create a storage account and an empty subnet for the Function App.

 

1. Create a storage account

A storage account is required in the same region as the Function App.  Create a storage account with the following configuration.

 

 

 

The other parameter tabs are not required for this example.

Click ‘Create’

 

1. Create an empty subnet

The function has to be able to reach the vCenter, NSXT and HCX appliances by IP address.  Function Apps require an empty subnet on a network that is connected to the AVS environment.

Create a subnet on an network that is able to reach the AVS environment and check to make sure there are no Firewalls or other restrictions in place.

In this example, on the vNet called Core-vNet, a subnet called AV-CC-FunctionSubnet has been created.  The minimum size is a /30.  Generally a larger subnet should be used if multiple Function Apps will use it.

 

              

 

 

 

5.      Initial Function App Build

Select Function App in the Azure portal then click on ‘+ Create’.  This will launch the Function App wizard.

 

1. Select a hosting plan:  Choose Functions Premium, it is the most cost effective plan that supports network integration, which is required for this function. Then click ‘Select’

 

 

 

 

1. Fill in the basics tab – we are using names specific to this lab, follow your own naming conventions.
2. Subscription where your SDDC exists
3. Resource Group                     “AVS-Canada-Central”
4. Function App name                     “AVS-CC-FunctionApp
5. Deploy code/container        “Code”
6. Runtime stack               “PowerShell Core”
7. Version                                  “7.4”  (minimum)
8. Region                     Canada-Central
9. Operating System               “Windows”
10. Windows Plan                     “CC-ASP-AVS”
11. Pricing plan*                     “Elastic Premium EP1”   (start small)
12. Zone redundancy               “Disabled”  (go to Enabled when moving to prod)

 

*Note that not all service plans are available in all regions.

 

 

 

 

Click Next: Storage

 

 

 

4. If a storage account was not created, you can create one here, otherwise, select the storage account completed above in Part 4.

Select ‘Configure basic diagnostics settings now” and select the Log Analytics Workspace identified in part 1.

 

Storage account:       avsccfunctionapp (v2)

LAW:                  AVSRelatedLogsOnly

 

 

 

Click Next: Networking

 

 

 

3. The Function App needs to be able to connect to a subnet that can reach the AVS management servers.  For this example we are not using private endpoints.  Select ‘Enable vNet Integration and then select the subnet connected in step 3.

 

 

 

Click Next: Monitoring

 

1. Under monitoring, set ‘Enable application insights’ to yes so that we can see the logs.  Create a new Application insights.  This is useful for troubleshooting.

 

 

 

 

 

 

And then use the Log Ananytics Workspace that was defined in the prerequisites or one that already exists

 

 

 

Click OK to close the ‘New Application Insights’ pane

Click Next:Deployment

 

1. Under deployment, leave this empty, this example will not use CI/DC or store any code in Github.  However, it is strongly recommended that you do use infrastructure as code to deploy this function if you have multiple  SDDCs.  (Each function app is available only to SDDCs located in the same region).

 

1. Click Next: Tags

 

1. Add any tags you want applied

Click Next: Review + Create , then when the validation is completed

 

 

 

Click:  Create 

 

Note:

There is an option beside the Create button to download a template for this function app.

 

 

 

1. Once the validation is complete and the build runs the following screen should be visible.

 

 

 

 

Click on “Go to resource”

 

CHECK FUNCTION NETWORK INTEGRATION (Outbound Internet Traffic is enabled – might need to disable – don’t know what this actually means).

 

1. Install the relevant modules (KeyVault, LogAnalytics, PowerCLI).

 

These are the modules that will load so that the function works with the various services and commands.

 

Click on the App Files and then using the pull-down, select ‘requirements.psd1’

 

 

 

 

 

Now insert the following text below line 7.  These are the PowerShell modules required to run the commands that are being called from the Function code.

 

'Az.KeyVault' = '5.3.0'

'Az.LogAnalytics' = '1.0.1'

'VMware.PowerCLI' = '13.2.1.22851661'

 

So that it looks like this:

 

 

 

Add in the az.vmware version 0.7.1  to this list

 

To check for the latest version, go to the www.powershellgallery.com.  This example looks for the PowerCLI latest verions, search for az.KeyVault and az.LogAnalytics as well.

 

 

 

 

 

1. Create a managed identity for the function so that it can access the KeyVault.

From the Function App portal page, click on Identity, turn the Status to On, and then click on the Save icon, and click Yes to enable it.

 

 

 

 

 

Now we need configure the ‘Azure role assignments’

 

 

 

 

Then click on the ‘+Add role assignments (Preview) and complete the assignment by selecting the Scope as KeyVault, the correct subscription, the KeyVault we are using (CC-AVS-KeyVault) and the Role of “KeyVault Secrets User.  This role is the minimum required permission for the system assigned identity to read the secret values.

 

 

 

 

The click Save

 

 

 

6.      Creating the Functions App common variable retrieval code

Now that the function app has been configured to support the function we want to execute, the functions themselves need to be defined.

1. From the Function App screen click on ‘Create function’

 

 

 

 

Select the ‘Timer trigger’ as this function, as most in this category of monitoring/alerting, will be run on a schedule.

 

 

 

Click Create.

 

 

On the template details screen, change the ‘Function name’ to match the function’s function.

Use the Schedule to determine how often the function should run.  Note the format is based on Cron job scheduling (see link in the ‘Appendix’).  In this case, the function is set to run every 5 minutes.

 

 

             Then click ‘Create’ at the bottom of the screen.

 

1. This will result in the following screen

 

 

Save this for now. Close the window using the  X  in the top right hand corner and select ‘Environment Variables’ from the menu on the left hand side.

 

 

 

1. On the Environment variables page, click on +Add and enter the following variables. For multi-region vCenters, use a more complex naming convention (ie when you have multiple SDDCs).

 

1. Name:              vCenterURL

Value:                https://10.4.0.2                (your vCenter instance IP address)

 

1. Name:              NSXTURL

Value:                https://10.4.0.3                (your NSX-T instance IP address)

 

1. Name: HCXRL

Value:                https://10.4.0.9                (your HCX instance IP address)

 

1. Name:              KeyVaultName

Value:                 CC-AVS-Keyvault             (name of the KeyVault created in the prereqs)

 

             Click on the +Add and enter the values above for each of the 4 variables.

 

 

 

 

 

 

 

 

 

 

 

The result should end up with app settings that look like this

 

 

Click ‘Apply’ to save the Environment variables.

 

1. Add the Environment variables and KeyVault secrets to the Get-vCenter-Data app.

Click on the Overview to show the functions page, and select the ‘CheckStorageTime’ function we created above.

 

 

Delete lines 4 to 14.

Step 1 – Add the Environment Variables

 

# Environment Variables

$LogType="AVSLogs"

$KeyVaultName=$env:KeyVaultName

$HCX-URL = $env:HCXURL

$vCenter-URL = $env:vCenterURL

$NSXT-URL = $env:NSXTURL

 

                           The result should look like this:

 

 

 

 

Step 2 – Add the KeyVault Variables

# KeyVault Secret Variables

$SharedKey=                               Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name                                                                           "sharedkey" -asplaintext

$avs-vCenterAdmin=                  Get-AzKeyVaultSecret -VaultName $KeyVaultName
                                                     -Name "avs-vCenterAdmin" -asplaintext

$avs-NSXTAdmin=                      Get-AzKeyVaultSecret -VaultName $KeyVaultName
                                                     -Name "avs-NSXTAdmin" -asplaintext

$avs-vCenterAdminPassword=              Get-AzKeyVaultSecret -VaultName $KeyVaultName
                                                     -Name " avs-vCenterAdminPassword " -asplaintext

$avs-NSXTAdminPassword=              Get-AzKeyVaultSecret -VaultName $KeyVaultName
                                                                   -Name " avs- NSXTAdminPassword " -asplaintext

 

The result should look like this:

 

 

 

Resources

Cron job format and time zone: https://cloud.google.com/scheduler/docs/configuring/cron-job-schedules

 

Troubleshooting

 

Variable retrievals

If you run a Code+Test, then lines 18 to 31 can be used to validate the environment variables and KeyVault secrets are being retrieved successfully.

 

 

Lines 33 to 40 are used to validate network connectivity between the function the and the IP address of the vCenter, HCX and NSXT virtual machines.

The results should look like this and show True:

 

 

 

Sample showing the code required to retrieve the variables from KeyVault:

 

1. $sharedKey=Get-AzKeyvaultSecret -VaultName $KeyVaultName -Name "SharedKey" -asplaintext
   
   
2. $avsVCenterAdmin=Get-AzKeyvaultSecret -VaultName $KeyVaultName -Name "avsVCenterAdmin" -asplaintext
3. $avsVCenterAdminPassword=Get-AzKeyvaultSecret -VaultName $KeyVaultName -Name "avsVCenterAdminPassword" -asplaintext
   
   
4. $avsNSXTAdmin= Get-AzKeyvaultSecret -VaultName $KeyVaultName -Name "avsNSXTAdmin" -asplaintext
5. $avsNSXTAdminPassword= Get-AzKeyvaultSecret -VaultName $KeyVaultName -Name "avsNSXTAdminPassword" -asplaintext

 

 

 

 

Example 1 - Diagnostics Available

This is a simple script to check that the diagnostics for the service is turned on.  I've left in lots of lines as comments that were used during troubleshooting and validation.  The output is to the AVS_Diagnostics table; which is automatically created the first time you run the script.

 

# Input bindings are passed in via param block.

param($Timer)

# Environment Variables

$LogType="AVS_DiagnosticsStatus"

$KeyVaultName=$env:KeyVaultName

$HCXURL = $env:HCXURL

$vCenterURL = $env:vCenterURL

$NSXTURL = $env:NSXTURL

$CustomerID = $env:LAW_WorkSpace_ID

# KeyVault Secrets

$SharedKey=Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name "avsLAWKey" -asplaintext

$avsvCenterAdmin=Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name "avsvCenterAdmin" -asplaintext

$avsNSXTAdmin=Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name "avsNSXTAdmin" -asplaintext

$avsvCenterAdminPassword=Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name "AVSSDDCvCentrerAdminAccountPassword" -asplaintext

$avsNSXTAdminPassword=Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name "avsNSXTAdminPassword" -asplaintext

$avsLAWSecret=Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name "avsLAWKey" -asplaintext

$avsTenantID=Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name "avsTenantID" -asplaintext

#Note this example uses the cloudadmin@vsphere.local account to authenticate with both vCenter and HCX

#Note this example uses cloudadmin account to authenticate with NSXT

# The following are used for checking that environment variables and keyvault secrets are being retrieved.

 "LAWSharedKey: $sharedKey"

 "CustomerID: $CustomerID"

# "vCenterURL: $vCenterURL"

# "HCXURL: $HCXURL"

# "NSXTURL: $NSXTURL"

# "KeyVault: $KeyVaultName"

# "LogAnalyticsKey:$SharedKey”

# “avsvCenterAdmin:$avsvCenterAdmin”

# “avsNSXTAdmin:$avsNSXTAdmin”

# “avsvCenterAdminPassword:$avsvCenterAdminPassword”

# “avsNSXTAdminPassword:$avsNSXTAdminPassword”

# The following are used to verify network connectivity to the vCenter, HCX and NSXT

# $vCenter=Test-connection $vCenterURL.trim("https://") -tcpport 443

# $VCntr=Test-connection $vCenterURL.trim("https://") -tcpport 443

# $HCX=Test-connection $HCXURL.trim("https://") -tcpport 443

# $NSXT=Test-connection $NSXTURL.trim("https://") -tcpport 443

# "VC Network connection valid: $vCntr"

# "HC Network connection valid: $HCX"

# Create Credential Automatically for NSXT (ENABLED)

# END of the variable definitions and authentication process - this is repeatable for all powershell and URI queries.

# ___________________________________________________________________________________________________________________

#Get list of private clouds

$ListOfSDDCs = Get-AzVmwarePrivateCloud

$result = @()

$counter = 0

ForEach ($SDDCName in $ListOfSDDCs.Name) {

   

    # Get the diagnostics setting for the cloud

    $RGName=$ListOfSDDCs[$counter].ResourceGroupName

    $SDDCName=$ListOfSDDCs[$counter].Name

 

    $counter += 1

    $SDDCResourceID = get-azresource -Resourcegroupname $RGName -resourcename $SDDCName | Select-Object -ExpandProperty ResourceId

    $DiagName =Get-AzDiagnosticSetting -ResourceId $SDDCResourceID | Select-Object -ExpandProperty Name

           

    if ($DiagName) {

        $LAWworkspace = Get-AzDiagnosticSetting -ResourceId $SDDCResourceID | Select-Object -ExpandProperty WorkspaceId

        $LAWworkspace = split-path $LAWworkspace -Leaf

        $Enabled = "Yes"

    } else {

       $Enabled = "No"

       $Diagname = "-"

       $LAWworkspace = "-"

    }

   

    $object = [PSCustomOBject]@{

        SDDC = $SDDCName

        Enabled = $Enabled

        LAW_Workspace = $LAWworkspace

        Diagnostics = $DiagName

    }

     $result += $object

}

# $result

#"___________________________________________________________"

 

$ResultData = $result | ConvertTo-Json -Depth 100

$ResultData

 Send-LogToLogAnalytics -CustomerId $CustomerId `

                         -SharedKey $sharedKey `

                         -Log $ResultData `

                         -LogType $LogType

 

 

Example 2:  vCenter Alerts

In this example we use the PowerCLI add on to pull all open alerts from vCenter.  Note again that I have left in a number of lines commented out that were used for troubleshooting.

 

# Input bindings are passed in via param block.

param($Timer)

# Environment Variables

$LogTable="TempTable"

$KeyVaultName=$env:KeyVaultName

$HCXURL = $env:HCXURL

$vCenterURL = $env:vCenterURL

$NSXTURL = $env:NSXTURL

$CustomerID = $env:LAW_WorkSpace_ID

# KeyVault Secrets

$SharedKey=Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name "avsLAWKey" -asplaintext

$avsvCenterAdmin=Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name "avsvCenterAdmin" -asplaintext

$avsNSXTAdmin=Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name "avsNSXTAdmin" -asplaintext

$avsvCenterAdminPassword=Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name "AVSSDDCvCentrerAdminAccountPassword" -asplaintext

$avsNSXTAdminPassword=Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name "avsNSXTAdminPassword" -asplaintext

$avsLAWSecret=Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name "avsLAWKey" -asplaintext

#Note this example uses the cloudadmin@vsphere.local account to authenticate with both vCenter and HCX

#Note this example uses cloudadmin account to authenticate with NSXT

# The following are used for checking that environment variables and keyvault secrets are being retrieved.

# "LAWSharedKey: $sharedKey"

# "CustomerID: $CustomerID"

# "vCenterURL: $vCenterURL"

# "HCXURL: $HCXURL"

# "NSXTURL: $NSXTURL"

# "KeyVault: $KeyVaultName"

# "LogAnalyticsKey:$SharedKey”

# “avsvCenterAdminPassword:$avsvCenterAdminPassword”

# “avsNSXTAdminPassword:$avsNSXTAdminPassword”

 Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false

# END of the variable definitions and authentication process - this is repeatable for all powershell and URI queries.

# ___________________________________________________________________________________________________________________

$vCenter = Connect-viserver $vCenterURL.trim('https://') -protocol https -user $avsvCenterAdmin -password $avsvCenterAdminPassword

function Get-TriggeredAlarm {

    [CmdletBinding()]

    param(

        [string]$VM,

        [string]$VMHost,

        [string]$Datacenter

    )

    BEGIN {

        switch ($PSBoundParameters.Keys) {

            'VM' {$entity = Get-VM -Name $VM -ErrorAction SilentlyContinue}

            'VMHost' {$entity = Get-VMHost -Name $VMHost -ErrorAction SilentlyContinue}

            'Datacenter' {$entity = Get-Datacenter -Name $Datacenter -ErrorAction SilentlyContinue}

            default {$entity = $null}

        }

        if ($null -eq $entity) {

            Write-Warning "No vSphere object found."

            break

        }

    }

    PROCESS {

        if ($entity.ExtensionData.TriggeredAlarmState -ne "") {

            $alarmOutput = @()

            foreach ($alarm in $entity.ExtensionData.TriggeredAlarmState) {

                $tempObj = "" | Select-Object -Property Entity, Alarm, AlarmStatus

                $tempObj.Entity = Get-View $alarm.Entity | Select-Object -ExpandProperty Name

                $tempObj.Alarm = Get-View $alarm.Alarm | Select-Object -ExpandProperty Info | Select-Object -ExpandProperty Name

                $tempObj.AlarmStatus = $alarm.OverallStatus

#                $tempObj.AlarmMoRef = $alarm.Alarm

#                $tempObj.EntityMoRef = $alarm.Entity

                $alarmOutput += $tempObj

            }

            $alarmOutput | Format-Table -AutoSize

        }

    }

    END {

        if ($entity -ne $null -and $entity.ExtensionData.TriggeredAlarmState -ne "") {

             #$jsonArray = $alarmOutput  | ConvertTo-Json -Depth 100

             #"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

             #Write-Output "JSON Array Output #1:"

             #Write-Output $jsonArray

             #"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

            return $alarmOutput | ConvertTo-Json -Depth 100

        }

    }

}

$result = Get-TriggeredAlarm -Datacenter "sddc-datacenter"

# Print the output

"_____________________________________________"

Write-Output "Result Output #2:"

Write-Output $result

"_____________________________________________"

 

Send-LogToLogAnalytics -CustomerId $CustomerId `

                         -SharedKey $sharedKey `

                         -Log $result `

                         -LogType "NewLogTable"

# Disconnect from vCenter Server

Disconnect-VIServer -Confirm:$false