Firstly, I would like to thank Benjamin Kovacevic and Yael Bergman for their help with this article.
While the existing Sentinel GCP Pub/Sub Audit Logs connector documented here provides a way to ingest GCP platform audit logs, ingesting GCP Firewall logs or VPS logs remains a needed capability.
In this blog post I will show a simple way to ingest Google Cloud GCP Firewall logs or VPS logs.
In order to do this ingestion, I have utilized the Log Ingestion API with PowerShell to accomplish this.
Note: The code provided in this post is just a sample provided AS-IS. Further code optimization and additions could be added as required.
Following is the summary of tasks that are required in order to accomplish this from a high level:
1- Create Microsoft Entra application
2- Create data collection endpoint
3- Create new table in Log Analytics workspace
4- Create a new service account in the GCP project with the needed assigned GCP IAM role.
5- Create a new GCP PUBSUB topic and a new GCP PULL typed PUBSUB subscription.
6- Construct a JWT header and acquire a JWT token
7- Pull the PUBSUB messages from the PUBSUB REST API
8- Send a message ack back and ingest the message content into Sentinel
Detailed Steps
1- Create Microsoft Entra application
Working with Log Ingestion API requires to register a new App in Entra ID and note down the TenantId, AppId, Secret.
Steps to create a new App registration is documented here for reference.
2- Create data collection endpoint
The DCE is required in order to receive the incoming data stream. Required steps to create a new DCE is documented here.
3- Create new table in Log Analytics workspace
Before you can send data to the workspace, you need to create the custom table where the data will be sent. Required steps to create a new table is documented here.
What is remaining up to this point is to take note of the DCR ImmutableId and also assign role Monitoring Metrics Publisher to DCR as described in details in same document referenced above.
Note that the new table schema has to match exactly every ingested property's. For instance I have used following column names when creating the new table schema
TimeGenerated
publishTime
messageId
insertId
dest_ip
dest_port
protocol
src_ip
src_port
disposition
project_id
region
vm_name
zone
direction
ip_protocol
priority
reference
project_id
subnetwork_name
subnetwork_id
vpc_name
logName
receiveTimestamp
Logtype
timestamp
4- Create a new service account in the GCP project with the needed assigned GCP IAM role.
5- Create a new GCP PUBSUB topic and a new GCP PULL typed PUBSUB subscription.
Here for both #4 and #5 we move to do some GCP side configurations. As a start we need to create a new service account which we are going to impersonate and this service account has to be assigned the correct IAM roles to enable it to pull and ack messages from the PUBSUB.
Steps to create a new service account and a new role in GCP in addition to creating a new PUBSUB topic and log sink are explained in details in this document.
Note: Here also note that it's recommended to create a filter to only have Firewall\VPC logs in this subscription while creating the sink.
6- Construct a JWT header and acquire a JWT token
Following this GCP authentication documentation here, we could construct a JWT header that we can use to acquire an access token.
7- Pull the PUBSUB messages from the PUBSUB REST API
To accomplish this step we could use the standard Method: projects.subscriptions.pull as documented here. Note that the ReceivedMessage in the response body contains an object called PubsubMessage where the data field is a Base64 encoded (PubsubMessage) string that will need to be decoded in the code before ingestion in Sentinel.
8- Send a message ack back and ingest the message content into Sentinel
Here an ack is recommended to be sent back in order to have that message removed from the PUBSUB subscription queue.
Now to the code work
- Putting some needed parameters upfront
$appId = "xxxxxxxxxxxxxxxxxxxxxxxxxx"
$appSecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$tenantId = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$ingestionuri = "https://xxxxxxxxxxxxxxxxx.westeurope-1.ingest.monitor.azure.com"
$CertFile = "C:\xxxx\sentinel.p12"
$CertPassword = "notasecret"
$Project = "xxxxxxxx"
$ServiceAccountName = "sentinelserviceaccount"
$ServiceAccount = "sentinelserviceaccount@xxxxxxxxxxxxxx.iam.gserviceaccount.com"
$Scope = "https://www.googleapis.com/auth/pubsub"
$ExpirationSeconds = 3600
- Function that constructs JWT header and acquire access token
function CreateJWT {
Write-Host "Attempting to obtain JWT access token"
# import certificate
$Certificate = [System.Security.Cryptography.X509Certificates.X509Certificate2]::new($CertFile,$CertPassword,[System.Security.Cryptography.X509Certificates.X509KeyStorageFlags]::Exportable)
$RSACryptoServiceProvider = New-Object System.Security.Cryptography.RSACryptoServiceProvider
$RSACryptoServiceProvider.ImportParameters($Certificate.PrivateKey.ExportParameters($true))
# create JWT Header
$JwtHeader = '{"alg":"RS256","typ":"JWT"}'
$JwtHeaderBase64 = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($JwtHeader))
$JwtHeaderBase64UrlEncoded = $JwtHeaderBase64 -replace "/","_" -replace "\+","-" -replace "=", ""
# create JWT Claim Set
$Now = (Get-Date).ToUniversalTime()
$NowUnixTimestamp = [Math]::Floor([decimal](Get-Date -Date $Now -UFormat "%s"))
$ExpirationUnixTimestamp = $NowUnixTimestamp + $ExpirationSeconds
$JwtClaimSet = @"
{
"iss":"$ServiceAccount",
"scope":"$Scope",
"aud":"https://oauth2.googleapis.com/token",
"exp":$ExpirationUnixTimestamp,
"iat":$NowUnixTimestamp
}
"@
$JwtClaimSetBase64 = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($JwtClaimSet))
$JwtClaimSetBase64UrlEncoded = $JwtClaimSetBase64 -replace "/","_" -replace "\+","-" -replace "=", ""
# calculate Signature
$StringToSign = $JwtHeaderBase64UrlEncoded + "." + $JwtClaimSetBase64UrlEncoded
$SHA256 = [System.Security.Cryptography.SHA256]::Create()
$Hash = $SHA256.ComputeHash([Text.Encoding]::UTF8.GetBytes($StringToSign))
$SignatureBase64 = [Convert]::ToBase64String($RSACryptoServiceProvider.SignData([System.Text.Encoding]::UTF8.GetBytes($StringToSign),"SHA256"))
$SignatureBase64UrlEncoded = $SignatureBase64 -replace "/","_" -replace "\+","-" -replace "=", ""
# create JWT
$Jwt = $JwtHeaderBase64UrlEncoded + "." + $JwtClaimSetBase64UrlEncoded + "." + $SignatureBase64UrlEncoded
# send JWT request for oauth access token
$Body = "grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=$Jwt"
$AccessToken = Invoke-RestMethod -Method Post -Uri "https://oauth2.googleapis.com/token" -ContentType "application/x-www-form-urlencoded" -Body $Body | Select-Object -ExpandProperty access_token
Write-Host "Access token has been obtained successfully" #; "-"*50
Write-Host " "
return $AccessToken
}
- Function that pulls messages from PUBSUB
function pull-messages {
Write-Host "Attempting to pull PUBSUB messages"
$Pulluri = "https://pubsub.googleapis.com/v1/projects/sentinel-403712/subscriptions/sentinel-sub:pull"
$PullBody = @{
maxMessages = "100"
}
$pulledmessages = Invoke-RestMethod -Method Post -Uri $Pulluri -Headers @{"Authorization"="Bearer $JWTToken"} -Body ($PullBody | ConvertTo-Json) -ContentType 'application/json'
Write-Host "PUBSUB Messages have been obtained successfully"
write-host " "
return $pulledmessages
}
- Function that sends back an acknowledgement
function ackmsg {
param (
[string]$ackidss
)
$AckBody = @{
"ackIds" = $ackidss
}
$ackuri = "https://pubsub.googleapis.com/v1/projects/xxxxxxxxxx/subscriptions/xxxxxxxxxxx:acknowledge"
$autoack = Invoke-RestMethod -Method Post -Uri $ackuri -Headers @{"Authorization"="Bearer $JWTToken"} -Body ($AckBody | ConvertTo-Json) -ContentType 'application/json'
}
- Function that do the actual ingestion into the new table
Note that more data processing is done in order to decode the base64 encoded data and also to do some string manipulation to have all properties ready in the right shape for the ingestion.
function ingestmsg{
param (
[PSCustomObject]$mmm
)
# $mmm
$mmm = $mmm -replace ";", ""
$mmm = $mmm -replace "}", ""
$msgobjj = $mmm | ConvertFrom-String -PropertyNames data, attributes,messageId,publishTime
$msgobjj.data = $msgobjj.data -replace "@{data=", ""
$msgobjj.attributes = $msgobjj.attributes -replace "attributes=", ""
$msgobjj.messageId = $msgobjj.messageId -replace "messageId=", ""
$msgobjj.publishTime = $msgobjj.publishTime -replace "publishTime=" , ""
$msgobjj.data = [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($msgobjj.data))
$TimeGenerated = Get-Date ([datetime]::UtcNow) -Format O
$data = $msgobjj.data | ConvertFrom-Json
$publishTime = $msgobjj.publishTime
$messageId = $msgobjj.messageId
$InsertId = $data.InsertId
$dest_ip = $data.jsonPayload.connection.dest_ip
$dest_port= $data.jsonPayload.connection.dest_port
$protocol = $data.jsonPayload.connection.protocol
$src_ip = $data.jsonPayload.connection.src_ip
$src_port = $data.jsonPayload.connection.src_port
$disposition = $data.jsonPayload.disposition
$project_id = $data.jsonPayload.instance.project_id
$region = $data.jsonPayload.instance.region
$vm_name = $data.jsonPayload.instance.vm_name
$zone = $data.jsonPayload.instance.zone
$direction = $data.jsonPayload.rule_details.direction
$ip_protocol = $data.jsonPayload.rule_details.ip_port_info.ip_protocol
$priority = $data.jsonPayload.rule_details.priority
$reference = $data.jsonPayload.rule_details.reference
$project_id = $data.jsonPayload.instance.project_id
$subnetwork_name = $data.resource.labels.subnetwork_name
$subnetwork_id = $data.resource.labels.subnetwork_id
$vpc_name = $data.jsonPayload.vpc.vpc_name
$logName = $data.logName
$timestamp = $data.timestamp
$type = $data.resource.type
$receiveTimestamp = $data.receiveTimestamp
$staticData = @"
[
{
"TimeGenerated": "$TimeGenerated",
"publishTime": "$publishTime",
"messageId": "$messageId",
"insertId": "$InsertId",
"dest_ip": "$dest_ip",
"dest_port": "$dest_port",
"protocol": "$protocol",
"src_ip": "$src_ip",
"src_port": "$src_port",
"disposition":"$disposition",
"project_id":"$project_id",
"region":"$region",
"vm_name":"$vm_name",
"zone":"$zone",
"direction":"$direction",
"ip_protocol":"$ip_protocol",
"priority":"$priority",
"reference":"$reference",
"project_id":"$project_id",
"subnetwork_name":"$subnetwork_name",
"subnetwork_id":"$subnetwork_id",
"vpc_name":"$vpc_name",
"logName":"$logName",
"receiveTimestamp":"$dtimestamp",
"Logtype":"$resource.type",
"timestamp":"$receiveTimestamp"
}
]
"@;
Add-Type -AssemblyName System.Web
$appId = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$appSecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$tenantId = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$scope= [System.Web.HttpUtility]::UrlEncode("https://monitor.azure.com//.default")
$body = "client_id=$appId&scope=$scope&client_secret=$appSecret&grant_type=client_credentials";
$headers = @{"Content-Type"="application/x-www-form-urlencoded"};
$uri = "https://login.microsoftonline.com/$tenantId/oauth2/v2.0/token"
$bearerToken = (Invoke-RestMethod -Uri $uri -Method "Post" -Body $body -Headers $headers).access_token
$dceEndpoint = "https://xxxxxxxxxxxxxxxxxx.westeurope-1.ingest.monitor.azure.com" #the endpoint property of the Data Collection Endpoint object
$dcrImmutableId = "dcr-xxxxxxxxxxxxxxxxxxxxxxxxxxxx" #the immutableId property of the DCR object
$streamName = "Custom-GcpFWLogs_CL" #name of the stream in the DCR that represents the destination table
$body = $staticData;
$headers = @{"Authorization"="Bearer $bearerToken";"Content-Type"="application/json"};
$urii = "$dceEndpoint/dataCollectionRules/$dcrImmutableId/streams/$($streamName)?api-version=2021-11-01-preview"
$uploadResponse = Invoke-RestMethod -Uri $urii -Method "Post" -Body $body -Headers $headers
}
- The actual code start running from here
$JWTToken = CreateJWT
$messages = pull-messages
$msgcount = ($messages.receivedMessages).count
Write-Host ""
write-host "Message count is:" $msgcount; ""
$msg = $messages.receivedMessages.message
if($msgcount -eq 1 ){
$ack = $messages.receivedMessages.ackid
write-host "Attempting to ack message by calling ack function"
ackmsg -ackidss $ack
Write-Host "message has been acked successfully"
$msg = $messages.receivedMessages.message
write-host "Attempting to ingest message by calling ingest function"
ingestmsg $msg
Write-Host "message has been ingested successfully"
}
<# Action when all if and elseif conditions are false #>
if($msgcount -eq 0 )
{
Write-Host "no new messages"
}
if($msgcount -gt 1 )
{
for ($i=1; $i -lt $msgcount; $i++)
{
$acks = $messages.receivedMessages.ackid
$acked = $acks[$i]
write-host "Attempting to ack message" $i "by calling ack function"
ackmsg -ackidss $acked
Write-Host "message" $i "has been acked successfully"
$msgs = $messages.receivedMessages.message
$msgg = $msgs[$i]
write-host "Attempting to ingest message" $i "by calling ingest function"
ingestmsg $msgg
Write-Host "message" $i "has been ingested successfully"
}
}
How data appears in Sentinel:
Final notes:
- This is a sample of code on how it could be done. This could be created in a form of Azure FunctionApp with a scheduled recurrence for instance.
- Some error handling could also be added for better monitoring of this ingestion pipeline.