Architecture Guidance: How to ingest GCP Firewall\VPC logs into Microsoft Sentinel
Published Nov 08 2023 10:02 AM 5,787 Views
Microsoft

Firstly, I would like to thank  Benjamin Kovacevic and Yael Bergman for their help with this article.

 

While the existing Sentinel GCP Pub/Sub Audit Logs connector documented here provides a way to ingest GCP platform audit logs, ingesting GCP Firewall logs or VPS logs remains a needed capability.

 

In this blog post I will show a simple way to ingest Google Cloud GCP Firewall logs or VPS logs.

 

In order to do this ingestion, I have utilized the Log Ingestion API with PowerShell to accomplish this. 

 

Note: The code provided in this post is just a sample provided AS-IS. Further code optimization and additions could be added as required.

 

Following is the summary of tasks that are  required in order to accomplish this from a high level:

 

1- Create Microsoft Entra application

2- Create data collection endpoint

3- Create new table in Log Analytics workspace

4- Create a new service account in the GCP project with the needed assigned GCP IAM role.

5- Create a new GCP PUBSUB topic and a new GCP PULL typed PUBSUB subscription.

6- Construct a JWT header and acquire a JWT token

7- Pull the PUBSUB messages from the PUBSUB REST API

8- Send a message ack back and ingest the message content into Sentinel

 

 

Detailed Steps

 

1- Create Microsoft Entra application

Working with Log Ingestion API requires to register a new App in Entra ID and note down the TenantId, AppId, Secret.

Steps to create a new App registration is documented here  for reference.

 

2- Create data collection endpoint

The DCE is required in order to receive the incoming data stream. Required steps to create a new DCE is documented here.

 

3- Create new table in Log Analytics workspace

Before you can send data to the workspace, you need to create the custom table where the data will be sent. Required steps to create a new table is documented here.

 

What is remaining up to this point is to take note of the DCR ImmutableId and also assign role Monitoring Metrics Publisher to DCR as described in details in same document referenced above.

 

Note that the new table schema has to match exactly every ingested property's. For instance I have used following column names when creating the new table schema

 

                    TimeGenerated

                    publishTime

                    messageId

                    insertId

                     dest_ip

                      dest_port

                     protocol

                     src_ip

                     src_port

                     disposition

                     project_id

                     region

                     vm_name

                     zone

                     direction

                    ip_protocol

                     priority

                     reference

                     project_id

                     subnetwork_name

                     subnetwork_id

                     vpc_name

                     logName

                    receiveTimestamp

                     Logtype

                     timestamp


4- Create a new service account in the GCP project with the needed assigned GCP IAM role.

 

5- Create a new GCP PUBSUB topic and a new GCP PULL typed PUBSUB subscription.

Here for both #4 and #5 we move to do some GCP side configurations. As a start we need to create a new service account which we are going to impersonate and this service account has to be assigned the correct IAM roles to enable it to pull and ack messages from the PUBSUB.

 

Steps to create a new service account and a new role in GCP in addition to creating a new PUBSUB topic and log sink are explained in details in this document.

 

Note: Here also note that it's recommended to create a filter to only have Firewall\VPC logs in this subscription while creating the sink.

 

6- Construct a JWT header and acquire a JWT token

Following this GCP authentication documentation here, we could construct a JWT header that we can use to acquire an access token.

 

7- Pull the PUBSUB messages from the PUBSUB REST API

To accomplish this step we could use the standard Method: projects.subscriptions.pull as documented here. Note that the ReceivedMessage  in the response body contains an object called PubsubMessage where the data field is a Base64 encoded (PubsubMessage)  string that will need to be decoded in the code before ingestion in Sentinel.

 

8- Send a message ack back and ingest the message content into Sentinel

Here an ack is recommended to be sent back in order to have that message removed from the PUBSUB subscription queue.

 

Now to the code work

 

  • Putting some needed parameters upfront

 

$appId = "xxxxxxxxxxxxxxxxxxxxxxxxxx"
$appSecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$tenantId = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

$ingestionuri = "https://xxxxxxxxxxxxxxxxx.westeurope-1.ingest.monitor.azure.com"
$CertFile = "C:\xxxx\sentinel.p12"
$CertPassword = "notasecret"
$Project = "xxxxxxxx"
$ServiceAccountName = "sentinelserviceaccount"
$ServiceAccount = "sentinelserviceaccount@xxxxxxxxxxxxxx.iam.gserviceaccount.com"
$Scope = "https://www.googleapis.com/auth/pubsub"
$ExpirationSeconds = 3600

 

 

  • Function that constructs JWT header and acquire access token

 

 


function CreateJWT {

Write-Host "Attempting to obtain JWT access token"
# import certificate
$Certificate = [System.Security.Cryptography.X509Certificates.X509Certificate2]::new($CertFile,$CertPassword,[System.Security.Cryptography.X509Certificates.X509KeyStorageFlags]::Exportable)
$RSACryptoServiceProvider = New-Object System.Security.Cryptography.RSACryptoServiceProvider
$RSACryptoServiceProvider.ImportParameters($Certificate.PrivateKey.ExportParameters($true))

# create JWT Header
$JwtHeader = '{"alg":"RS256","typ":"JWT"}'
$JwtHeaderBase64 = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($JwtHeader))
$JwtHeaderBase64UrlEncoded = $JwtHeaderBase64 -replace "/","_" -replace "\+","-" -replace "=", ""

# create JWT Claim Set
$Now = (Get-Date).ToUniversalTime()
$NowUnixTimestamp = [Math]::Floor([decimal](Get-Date -Date $Now -UFormat "%s"))
$ExpirationUnixTimestamp = $NowUnixTimestamp + $ExpirationSeconds
$JwtClaimSet = @"
{
    "iss":"$ServiceAccount",
    "scope":"$Scope",
    "aud":"https://oauth2.googleapis.com/token",
    "exp":$ExpirationUnixTimestamp,
    "iat":$NowUnixTimestamp
}
"@
$JwtClaimSetBase64 = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($JwtClaimSet))
$JwtClaimSetBase64UrlEncoded = $JwtClaimSetBase64 -replace "/","_" -replace "\+","-" -replace "=", ""

# calculate Signature
$StringToSign = $JwtHeaderBase64UrlEncoded + "." + $JwtClaimSetBase64UrlEncoded
$SHA256 = [System.Security.Cryptography.SHA256]::Create()
$Hash = $SHA256.ComputeHash([Text.Encoding]::UTF8.GetBytes($StringToSign))
$SignatureBase64 = [Convert]::ToBase64String($RSACryptoServiceProvider.SignData([System.Text.Encoding]::UTF8.GetBytes($StringToSign),"SHA256"))
$SignatureBase64UrlEncoded = $SignatureBase64 -replace "/","_" -replace "\+","-" -replace "=", ""

# create JWT
$Jwt = $JwtHeaderBase64UrlEncoded + "." + $JwtClaimSetBase64UrlEncoded + "." + $SignatureBase64UrlEncoded

# send JWT request for oauth access token
$Body = "grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=$Jwt"
$AccessToken = Invoke-RestMethod -Method Post -Uri "https://oauth2.googleapis.com/token" -ContentType "application/x-www-form-urlencoded" -Body $Body | Select-Object -ExpandProperty access_token

Write-Host "Access token has been obtained successfully" #; "-"*50
Write-Host " "
return $AccessToken
}

 

 

  • Function that pulls messages from PUBSUB

 

 



function pull-messages {
    Write-Host "Attempting to pull PUBSUB messages"
$Pulluri = "https://pubsub.googleapis.com/v1/projects/sentinel-403712/subscriptions/sentinel-sub:pull"
$PullBody = @{    
    maxMessages = "100"
    } 
$pulledmessages = Invoke-RestMethod -Method Post -Uri $Pulluri -Headers @{"Authorization"="Bearer $JWTToken"} -Body ($PullBody | ConvertTo-Json) -ContentType 'application/json'

Write-Host "PUBSUB Messages have been obtained successfully" 
write-host " "
return $pulledmessages
}

 

 

  • Function that sends back an acknowledgement

 

 


function ackmsg {
    

    param (
        [string]$ackidss
    )
   $AckBody = @{    
        "ackIds" = $ackidss
            } 
    $ackuri = "https://pubsub.googleapis.com/v1/projects/xxxxxxxxxx/subscriptions/xxxxxxxxxxx:acknowledge"
    $autoack = Invoke-RestMethod -Method Post -Uri $ackuri -Headers @{"Authorization"="Bearer $JWTToken"} -Body ($AckBody | ConvertTo-Json) -ContentType 'application/json'
    
}

 

 

  • Function that do the actual ingestion into the new table

Note that more data processing is done in order to decode the base64 encoded data and also to do some string manipulation to have all properties ready in the right shape for the ingestion.

 

 


function ingestmsg{
    
    param (
        [PSCustomObject]$mmm

    )

    # $mmm
    $mmm = $mmm -replace ";", ""
    $mmm = $mmm -replace "}", ""
    $msgobjj = $mmm | ConvertFrom-String -PropertyNames data, attributes,messageId,publishTime
    
   
    $msgobjj.data = $msgobjj.data -replace "@{data=", ""
    $msgobjj.attributes = $msgobjj.attributes -replace "attributes=", ""
    $msgobjj.messageId = $msgobjj.messageId -replace "messageId=", ""
    $msgobjj.publishTime = $msgobjj.publishTime -replace "publishTime=" , ""
    
    $msgobjj.data = [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($msgobjj.data))



$TimeGenerated = Get-Date ([datetime]::UtcNow) -Format O
$data = $msgobjj.data | ConvertFrom-Json
$publishTime = $msgobjj.publishTime
$messageId = $msgobjj.messageId
$InsertId = $data.InsertId
$dest_ip = $data.jsonPayload.connection.dest_ip
$dest_port= $data.jsonPayload.connection.dest_port
$protocol = $data.jsonPayload.connection.protocol
$src_ip = $data.jsonPayload.connection.src_ip
$src_port = $data.jsonPayload.connection.src_port
$disposition = $data.jsonPayload.disposition
$project_id = $data.jsonPayload.instance.project_id
$region = $data.jsonPayload.instance.region
$vm_name = $data.jsonPayload.instance.vm_name
$zone = $data.jsonPayload.instance.zone
$direction = $data.jsonPayload.rule_details.direction
$ip_protocol = $data.jsonPayload.rule_details.ip_port_info.ip_protocol
$priority = $data.jsonPayload.rule_details.priority
$reference = $data.jsonPayload.rule_details.reference
$project_id = $data.jsonPayload.instance.project_id
$subnetwork_name = $data.resource.labels.subnetwork_name
$subnetwork_id = $data.resource.labels.subnetwork_id
$vpc_name = $data.jsonPayload.vpc.vpc_name
$logName = $data.logName
$timestamp = $data.timestamp
$type  = $data.resource.type
$receiveTimestamp = $data.receiveTimestamp


$staticData = @"
              [
                {
                   
                    "TimeGenerated": "$TimeGenerated",
                    "publishTime": "$publishTime",
                    "messageId": "$messageId",
                    "insertId": "$InsertId",
                     "dest_ip": "$dest_ip",
                      "dest_port": "$dest_port",
                     "protocol": "$protocol",
                     "src_ip": "$src_ip",   
                     "src_port":  "$src_port",
                     "disposition":"$disposition",
                     "project_id":"$project_id",
                     "region":"$region",
                     "vm_name":"$vm_name",
                     "zone":"$zone",
                     "direction":"$direction",
                     "ip_protocol":"$ip_protocol",
                     "priority":"$priority",
                     "reference":"$reference",
                     "project_id":"$project_id",
                     "subnetwork_name":"$subnetwork_name",
                     "subnetwork_id":"$subnetwork_id",
                     "vpc_name":"$vpc_name",
                     "logName":"$logName",
                     "receiveTimestamp":"$dtimestamp",
                     "Logtype":"$resource.type",
                     "timestamp":"$receiveTimestamp"
                }

]
"@;
Add-Type -AssemblyName System.Web

$appId = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$appSecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$tenantId = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"


$scope= [System.Web.HttpUtility]::UrlEncode("https://monitor.azure.com//.default")   
$body = "client_id=$appId&scope=$scope&client_secret=$appSecret&grant_type=client_credentials";
$headers = @{"Content-Type"="application/x-www-form-urlencoded"};
$uri = "https://login.microsoftonline.com/$tenantId/oauth2/v2.0/token"
$bearerToken = (Invoke-RestMethod -Uri $uri -Method "Post" -Body $body -Headers $headers).access_token


$dceEndpoint = "https://xxxxxxxxxxxxxxxxxx.westeurope-1.ingest.monitor.azure.com" #the endpoint property of the Data Collection Endpoint object
$dcrImmutableId = "dcr-xxxxxxxxxxxxxxxxxxxxxxxxxxxx" #the immutableId property of the DCR object
$streamName = "Custom-GcpFWLogs_CL" #name of the stream in the DCR that represents the destination table


$body = $staticData;
$headers = @{"Authorization"="Bearer $bearerToken";"Content-Type"="application/json"};
$urii = "$dceEndpoint/dataCollectionRules/$dcrImmutableId/streams/$($streamName)?api-version=2021-11-01-preview"

$uploadResponse = Invoke-RestMethod -Uri $urii -Method "Post" -Body $body -Headers $headers

}

 

 

  • The actual code start running from here

 

 



$JWTToken = CreateJWT
$messages = pull-messages
$msgcount = ($messages.receivedMessages).count
Write-Host ""
write-host "Message count is:" $msgcount; ""
$msg = $messages.receivedMessages.message
if($msgcount -eq 1 ){
 $ack = $messages.receivedMessages.ackid
 write-host "Attempting to ack message by calling ack function"
 ackmsg -ackidss $ack
 Write-Host "message has been acked successfully"
 $msg = $messages.receivedMessages.message
 write-host "Attempting to ingest message by calling ingest function"
    ingestmsg $msg

    Write-Host "message has been ingested successfully"

}


    <# Action when all if and elseif conditions are false #>
    if($msgcount -eq 0 )
    {
        Write-Host "no new messages"
    }



    
    if($msgcount -gt 1  )
    {
        
for ($i=1; $i -lt $msgcount; $i++)
{
   $acks = $messages.receivedMessages.ackid
   $acked = $acks[$i]

   write-host "Attempting to ack message" $i "by calling ack function"

   ackmsg -ackidss $acked

   Write-Host "message" $i "has been acked successfully"

   $msgs = $messages.receivedMessages.message
   $msgg = $msgs[$i]

   write-host "Attempting to ingest message" $i "by calling ingest function"
   ingestmsg $msgg

   Write-Host "message" $i "has been ingested successfully"
}


    }

 

 

How data appears in Sentinel:

 

asasaUntitled.png

 

Final notes:

  • This is a sample of code on how it could be done. This could be created in a form of Azure FunctionApp with a scheduled recurrence for instance.
  • Some error handling could also be added for better monitoring of this ingestion pipeline.
Version history
Last update:
‎Nov 09 2023 01:39 AM
Updated by: