Scaling Up Syslog CEF Collection

Published Feb 20 2020 04:13 PM 10.9K Views

This blog post is authored by Nicholas DiCola

 

In the last few months working on Azure Sentinel, many customers have asked me about scaling up syslog CEF collection for getting data into Azure Sentinel.  I have created two sample architectures with code deployment for this purpose.  The samples is available at:

 

CEF-VMSS is for deploying native Azure Sentinel CEF collection by sending syslog CEF message to rsyslog which then sends the messages to the Log Analytics Agent.

 

Logstash-VMSS is for deploying Logstash on the VMs to do message manipulation which then sends the messages to the Log Analytics Agent.  You may also want to use this architecture and change the input to a source like Kafka.

 

I will not deep dive on all the topics of this architecture.  You can research each on your own and will focus on an overview of the architecture.

 

Virtual Machine Scale Set

The architecture starts with a VMSS which lets you manage and create a group of virtual machines.  These VMs can autoscale up and down additional instances based on schedule or demand.  The sample uses autoscale settings to configure the VMSS to scale up and down based on CPU (demand) of messages being sent.

 

I have included a Load Balancer in-front of the VMSS which will allow you to configure 1 destination IP address (the Public Ip Address) and it will spread the incoming messages across the running instances.

 

There are 2 ARM templates for RedHat and Unbuntu.  The templates deploy everything needed for the architecture.  One key part of the ARM templates is using cloud init to configure the VMSS instances as they are created.  Below is the Unbuntu cloud init files.

 

Cloud Init for CEF-VMSS:

#cloud-config
package_upgrade: true
runcmd:
  - sudo apt-get update
  - sudo wget https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/CEF/cef_installer.py&&sudo python cef_installer.py

As you can see the cloud init, it installs updates and the Log Analytics Agent using the Azure Sentinel CEF script.  The ARM template appends the workspace id and workspace key to the last line so that the agent gets connected to the right workspace

 

Cloud Init for Logstash-VMSS:

#cloud-config
package_upgrade: true
packages:
  - default-jre
runcmd:
  - wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
  - sudo apt-get install apt-transport-https
  - echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
  - sudo apt-get update
  - sudo apt-get install logstash
  - sudo /usr/share/logstash/bin/logstash-plugin install logstash-output-syslog
  - sudo /usr/share/logstash/bin/logstash-plugin update
  - wget -q https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Logstash-VMSS/logstash.config -O /etc/logstash/config.d/logstash.config
  - echo "update this line with wget -q https://sourceURL -O /etc/logstash/pipelines.yml if you have a custom pipelines file"
  - sudo systemctl start logstash.service
  - sudo wget https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/CEF/cef_installer.py&&sudo python cef_installer.py

It installs Java, Logstash, Logstash Syslog Output plugin and the Log Analytics Agent using the Azure Sentinel CEF script.  The ARM template appends the workspace id and workspace key to the last line so that the agent gets connected to the right workspace.

CEF

CEF is our default way to collect external solutions like firewalls and proxies.  The CEF install script will install the Log Analytics agent, configure rsyslog, and configure the agent for CEF collection.

Logstash

Logstash dynamically ingests, transforms, and ships your data regardless of format or complexity.  It has many input, filter and output plugins.  This can allow you to get data from many sources, manipulate the event data and output to the Log Analytics Agent locally on the machine.  There are many input plugins so this makes it easy to connect to other sources like Kafka.

 

Here is the sample logstash.conf file that is used in the sample architecture:

input {
  tcp {
    port => 5514
    type => syslog
    codec => cef
  }
  udp {
    port => 5514
    type => syslog
    codec => cef
  }
}

filter {
  geoip {
    source => "src"
    target => "srcGeoIP"
    add_field => { "sourceLongitude" => "%{[srcGeoIP][longitude]}" }
    add_field => { "sourceLatitude" => "%{[srcGeoIP][latitude]}" }
  }
  geoip{
    source => "dst"
    target => "dstGeoIP"
    add_field => { "destinationLongitude" => "%{[dstGeoIP][longitude]}" }
    add_field => { "destinationLatitude" => "%{[dstGeoIP][latitude]}" }
  }
  mutate{
    add_field => { "agentReceiptTime" => "%{@timestamp}" }
  }
}

output {
  syslog {
  host => "127.0.0.1"
  port => 25226
  protocol "tcp"
  codec => cef {
    reverse_mapping => true
    delimiter => "\r\n"
    vendor      => "%{deviceVendor}"
    product     => "%{deviceProduct}"
    version     => "%{deviceVersion}"
    signature   => "%{deviceEventClassId}"
    name        => "%{name}"
    severity    => "%{severity}"
    fields => [
      "deviceAction",
      "applicationProtocol",
      "deviceCustomIPv6Address1",
      "deviceCustomIPv6Address1Label",
      "deviceCustomIPv6Address2",
      "deviceCustomIPv6Address2Label",
      "deviceCustomIPv6Address3",
      "deviceCustomIPv6Address3Label",
      "deviceCustomIPv6Address4",
      "deviceCustomIPv6Address4Label",
      "deviceEventCategory",
      "deviceCustomFloatingPoint1",
      "deviceCustomFloatingPoint1Label",
      "deviceCustomFloatingPoint2",
      "deviceCustomFloatingPoint2Label",
      "deviceCustomFloatingPoint3",
      "deviceCustomFloatingPoint3Label",
      "deviceCustomFloatingPoint4",
      "deviceCustomFloatingPoint4Label",
      "deviceCustomNumber1",
      "deviceCustomNumber1Label",
      "deviceCustomNumber2",
      "deviceCustomNumber2Label",
      "deviceCustomNumber3",
      "deviceCustomNumber3Label",
      "baseEventCount",
      "deviceCustomString1",
      "deviceCustomString1Label",
      "deviceCustomString2",
      "deviceCustomString2Label",
      "deviceCustomString3",
      "deviceCustomString3Label",
      "deviceCustomString4",
      "deviceCustomString4Label",
      "deviceCustomString5",
      "deviceCustomString5Label",
      "deviceCustomString6",
      "deviceCustomString6Label",
      "destinationHostName",
      "destinationMacAddress",
      "destinationNtDomain",
      "destinationProcessId",
      "destinationUserPrivileges",
      "destinationProcessName",
      "destinationPort",
      "destinationAddress",
      "destinationUserId",
      "destinationUserName",
      "deviceAddress",
      "deviceHostName",
      "deviceProcessId",
      "endTime",
      "fileName",
      "fileSize",
      "bytesIn",
      "bytesOut",
      "eventOutcome",
      "transportProtocol",
      "requestUrl",
      "deviceReceiptTime",
      "sourceHostName",
      "sourceMacAddress",
      "sourceNtDomain",
      "sourceProcessId",
      "sourceUserPrivileges",
      "sourceProcessName",
     "sourcePort",
      "sourceAddress",
      "startTime",
      "sourceUserId",
      "sourceUserName",
      "agentHostName",
      "agentReceiptTime",
      "agentType",
      "agentId",
      "cefVersion",
      "agentAddress",
      "agentVersion",
      "agentTimeZone",
      "destinationTimeZone",
      "sourceLongitude",
      "sourceLatitude",
      "destinationLongitude",
      "destinationLatitude",
      "categoryDeviceType",
      "managerReceiptTime",
      "agentMacAddress"
      ]
    }
  }
}

The inputs accept both TCP and UDP on port 5514.  I used 5514 because Logstash runs as non-root and requires special configuration to use port 514.  I decided to keep it simple.  On input, its expecting CEF format using “codec => cef” and tags the event as syslog.

 

Once the event is accepted, I have added a few filters.  The first uses the GeoIP plugin which uses the local GeoLite2 database to lookup the source and destination IP addresses.  These are added to a custom field and to align with RFC compliance I then add_field to bring the latitude and longitude into proper fields.  I also mutate to copy the message received time into agentRecievedTime.  This is important as Logstash will send the message to the Log Analytics using its time which will end up as TimeGenerated in Log Analytics.  Doing this will allow you to see the original send time and the time Logstash sent it.  A simple compare will show you how long its take to process.

 

In the output section, I use the syslog plugin to output the message to the agent.  The agent listens on TCP 127.0.0.1:25226.  I set the output plugin to the CEF codec again and there are couple of key important configs.  “reverse_mapping => true” ensures that the message is sent using the short names (src vs sourceAddress) which is required by Log Analytics.  The fields portion requires all fields you want to send. I have included all fields the CEF codec supports.  If the fields doesn’t exist it wont be sent.

Azure Sentinel

Once the data is sent to the agent, it will follow all the normal CEF collection ingestion process and end up in CommonSecurityLog.  You can monitor the VMSS event per second using the following query:

CommonSecurityLog
| where _TimeReceived > ago(20m)
| summarize count() by bin(_TimeReceived, 1m), _ResourceId
| extend count =count_ /60
| sort by _TimeReceived desc

This will get all logs in the last 20m and summarize by TimeRecieved and ResourceId.  This gives you the # of event per minute, so you need to create a count column equal to count_ divided by 60 seconds.  Now you can see the EPS per VMSS instance.

If you have performance issues, I recommend you look into Ryslog performance or Logstash performance tuning.

 

Some future improvements I might add:

  • Implement impstats for rsyslog and send data to Log Analytics.  This would allow performance monitoring of rsyslog (dashboarding, queries)
  • Implement GeoIP in rsyslog
  • Implement Logstash monitoring APIs and send data to Log Analytics.  This would allow performance monitoring of Logstash (dashboarding, queries)
  • Create additional sample using fluentd
  • Create additional sample using syslog-ng

Thanks for reading!

12 Comments
Visitor

I moved this feedback to the github repo

Senior Member
Hi Nicholas, I was wondering if you know how I can daisy chain collectors like in the timed youtube link (https://youtu.be/_mm3GNwPBHU?list=PLOhMGpMOPKRHPHCvzia3EE5OY5EkQRCuH&t=896) was presented by one of Microsoft's Sentinel guys called Ofer. I was able to install the CEF-Syslog Ubuntu server on-prem but I am trying to do the Syslog-Collector proxy. I believe there must be some configs that need to change on both the nodes. Simply the architecture I was to put is such: 1. Install a CEF-Syslog on-prem Ubuntu machine to collect all CEF and Syslog sources 2. Forward the data from the CEF-Syslog machine to another Syslog-Collector-Proxy in another segment of my network to forward to Azure to Sentinel Can you help? Thank you, Egal
New Contributor

I see Standard_B16ms sku is used in the vmss.

Was this a decision or just by sample?

Hi @eciruam I actually need to update the template.  I worked with the agent team and they did some testing.  With an F4s_v2 VM size they were able to achieve higher EPS and recommended that as the size.

Senior Member

Nicholas,

 

do you have any concerns about scaling down a vm scale set without knowing if it's queue is empty?  isn't there some risk of lost messages?

 

I am toying with the idea of adapting my syslog-ng front-ends to push messages into a service bus queue (or multiple queues), and then using a custom input into logstash to peek a message from the queue, process it through logstash, and then deque the message after processing.

 

That would remove the need for load balancers, and allow the scale sets to expand/contract based on load (or ideally service bus queue sizes), and guarantee delivery of the messages.

 

thoughts?

@Justin Ainsworth 

theoretically, yea i guess that could happen.  but you could set autoscale settings to check network out instead of CPU.  https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-autosca...

 

yea you could do some kind of bus in azure.  i went the simple route to solve a customer request.

Senior Member

Hi Nicolas,

 

If someone's organisation policy does not allow SSH and public IP on load balancer, what is work around for this situation.

Established Member

Hi Nicholas DiCola (SECURITY JEDI) 

 

I have deployed the solution using the template, thank you for putting the work in to make it happen.

 

I am using the Ubuntu script.

 

I note when a VM is created the syslog service is not listening.  If I execute the cef_installer.py with workstation id and password from the root folder it reinstalls/reconfigures and syslog service is listening.

 

I am doing nothing different to the installer but getting a different outcome.  

 

Can you suggest anything to allow this to not require the human intervention?

 

Thank you,

George

Visitor

Hi Nicholas,

 

My Question might be somewhat in general.

1.) What types of data formats (Syslog, CEF, custom) can be taken up by Logstash for which it has some kind of mapping mechanism to ingest the data into Log Analytics workspace.

2.) Does Logstash have any mapping mechanism to map data format from any data source to convert into CEF or syslog which I suppose are the preferred choices of Sentinel?

3.) Do we really need Log Analytics Agent in between Logstash and Log Analytics workspace of sentinel?

4.) Which would be the best choice of data format (syslog, CEF, or custom) out from logstash to ingest into Log Analytics workspace of sentinel?

 

5.) Can the logstash-vmss be deployed on-prem?

 

6.) Could you please suggest to me the best choices of data intake format (I do understand various data sources may have their own data formats ) to Logstash from any data sources and data output format from Logstash to Log Analytics workspace of sentinel?

 

Regards,

Simranjeet

 

 

 

 

1.) What types of data formats (Syslog, CEF, custom) can be taken up by Logstash for which it has some kind of mapping mechanism to ingest the data into Log Analytics workspace.

Logstash is an event pipeline system.  it has many input plugins https://www.elastic.co/guide/en/logstash/current/input-plugins.html

It can also transform data during parsing.  So to answer many data formats can be changed to match CEF output.  This VMSS solution was designed specific to solve customer need to get SYSLOG (CEF FORMAT) messages, add geo ip information, then send to log analytics.

2.) Does Logstash have any mapping mechanism to map data format from any data source to convert into CEF or syslog which I suppose are the preferred choices of Sentinel?

Yes please review on Logstash documentation.

3.) Do we really need Log Analytics Agent in between Logstash and Log Analytics workspace of sentinel?

For this scenario, yes.  CommonEventLogs come from the agent only.  You could change the output to Custom Logs using the output connector for Log Analytics.

4.) Which would be the best choice of data format (syslog, CEF, or custom) out from logstash to ingest into Log Analytics workspace of sentinel?

There is no simple answer.  Depends on the data source.

 5.) Can the logstash-vmss be deployed on-prem?

Logstash can be deployed anywhere.  So can the Log A Agent.  The VMSS is specific to Azure resource.  You would need to write your own installer scripts for on-prem

 6.) Could you please suggest to me the best choices of data intake format (I do understand various data sources may have their own data formats ) to Logstash from any data sources and data output format from Logstash to Log Analytics workspace of sentinel?

It depends on your data source?  Is it a network Appliance like a firewall, normally that’s CEF.  Windows Event Logs should go to windows event or security event.

Visitor

Hi Nichola

I am grateful for the replies to my previous question.

Since you have told me that Logstash and Log A Agent can be deployed anywhere and I have seen on this page https://docs.microsoft.com/en-us/azure/sentinel/connect-logstash how to configure Logstash and send logs Log Analytics workspace using a log A output plugin.

I wish to know: 

 

1.) In the above link, there is no mention of Log A Agent which should be there in the case of Logstash-vmss's architecture. As per my understanding, the Log A output plugin takes logs from Logstash and ingests them into Log A workspace in a custom table which I suppose is neither syslog no cef format. Am I correct at this point?

 

2.) Can we build such a model where the whole of Microsoft's grand list of data sources(https://techcommunity.microsoft.com/t5/azure-sentinel/azure-sentinel-the-connectors-grand-cef-syslog...)  be ingested into sentinel in one single common format (cef/syslog not custom) using Logstash?

 

3.) I read somewhere sentinel gives better monitoring, analytics, correlation, incident generation, etc. if data from all data sources be ingested into sentinel in cef/syslog format. There can be quality issues if every data source has its own custom data format and table in sentinel which will not allow sentinel to do better analytics on data because of the randomness of data field names. Am I correct on this point?

 

4.) Can sentinel perform data correlation and analytics if there are N numbers of custom tables present for different data source security appliances?

 

Nicholas, I am sorry if I am eating your mind too much.

 

Regards,

Simran

 

1.) In the above link, there is no mention of Log A Agent which should be there in the case of Logstash-vmss's architecture. As per my understanding, the Log A output plugin takes logs from Logstash and ingests them into Log A workspace in a custom table which I suppose is neither syslog no cef format. Am I correct at this point?

the document is how to connect logstash to send custom logs.  The VMSS is to send syslog to CEF table in log a.  you can use either option, but the VMSS was very specific to get CEF logs into CEF table.

 

2.) Can we build such a model where the whole of Microsoft's grand list of data sources(https://techcommunity.microsoft.com/t5/azure-sentinel/azure-sentinel-the-connectors-grand-cef-syslog...)  be ingested into sentinel in one single common format (cef/syslog not custom) using Logstash?

Yes you could.

 

3.) I read somewhere sentinel gives better monitoring, analytics, correlation, incident generation, etc. if data from all data sources be ingested into sentinel in cef/syslog format. There can be quality issues if every data source has its own custom data format and table in sentinel which will not allow sentinel to do better analytics on data because of the randomness of data field names. Am I correct on this point?

Correct.  CEF is a standard format so running queries is much easier when all syslog data is in the same format.  if each has its own custom log then you need to write queries for each custom source.

 

4.) Can sentinel perform data correlation and analytics if there are N numbers of custom tables present for different data source security appliances?

Yes but it requires a more complex query.  Hence using CEF makes it easier.

%3CLINGO-SUB%20id%3D%22lingo-sub-1185854%22%20slang%3D%22en-US%22%3EScaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1185854%22%20slang%3D%22en-US%22%3E%3CP%3E%3CEM%3EThis%20blog%20post%20is%20authored%20by%20%3C%2FEM%3E%3CA%20href%3D%22https%3A%2F%2Ftwitter.com%2FMasterSecJedi%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3E%3CEM%3ENicholas%20DiCola%3C%2FEM%3E%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20the%20last%20few%20months%20working%20on%20Azure%20Sentinel%2C%20many%20customers%20have%20asked%20me%20about%20scaling%20up%20syslog%20CEF%20collection%20for%20getting%20data%20into%20Azure%20Sentinel.%26nbsp%3B%20I%20have%20created%20two%20sample%20architectures%20with%20code%20deployment%20for%20this%20purpose.%26nbsp%3B%20The%20samples%20is%20available%20at%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FAzure%2FAzure-Sentinel%2Ftree%2Fmaster%2FDataConnectors%2FCEF-VMSS%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Fgithub.com%2FAzure%2FAzure-Sentinel%2Ftree%2Fmaster%2FDataConnectors%2FCEF-VMSS%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FAzure%2FAzure-Sentinel%2Ftree%2Fmaster%2FDataConnectors%2FLogstash-VMSS%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Fgithub.com%2FAzure%2FAzure-Sentinel%2Ftree%2Fmaster%2FDataConnectors%2FLogstash-VMSS%3C%2FA%3E%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ECEF-VMSS%20is%20for%20deploying%20native%20Azure%20Sentinel%20CEF%20collection%20by%20sending%20syslog%20CEF%20message%20to%20rsyslog%20which%20then%20sends%20the%20messages%20to%20the%20Log%20Analytics%20Agent.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ELogstash-VMSS%20is%20for%20deploying%20Logstash%20on%20the%20VMs%20to%20do%20message%20manipulation%20which%20then%20sends%20the%20messages%20to%20the%20Log%20Analytics%20Agent.%26nbsp%3B%20You%20may%20also%20want%20to%20use%20this%20architecture%20and%20change%20the%20input%20to%20a%20source%20like%20Kafka.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EI%20will%20not%20deep%20dive%20on%20all%20the%20topics%20of%20this%20architecture.%26nbsp%3B%20You%20can%20research%20each%20on%20your%20own%20and%20will%20focus%20on%20an%20overview%20of%20the%20architecture.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%3EVirtual%20Machine%20Scale%20Set%3C%2FH2%3E%0A%3CP%3EThe%20architecture%20starts%20with%20a%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fvirtual-machine-scale-sets%2Foverview%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3EVMSS%3C%2FA%3E%20which%20lets%20you%20manage%20and%20create%20a%20group%20of%20virtual%20machines.%26nbsp%3B%20These%20VMs%20can%20autoscale%20up%20and%20down%20additional%20instances%20based%20on%20schedule%20or%20demand.%26nbsp%3B%20The%20sample%20uses%20autoscale%20settings%20to%20configure%20the%20VMSS%20to%20scale%20up%20and%20down%20based%20on%20CPU%20(demand)%20of%20messages%20being%20sent.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EI%20have%20included%20a%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fload-balancer%2Fload-balancer-overview%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3ELoad%20Balancer%3C%2FA%3E%20in-front%20of%20the%20VMSS%20which%20will%20allow%20you%20to%20configure%201%20destination%20IP%20address%20(the%20Public%20Ip%20Address)%20and%20it%20will%20spread%20the%20incoming%20messages%20across%20the%20running%20instances.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThere%20are%202%20ARM%20templates%20for%20RedHat%20and%20Unbuntu.%26nbsp%3B%20The%20templates%20deploy%20everything%20needed%20for%20the%20architecture.%26nbsp%3B%20One%20key%20part%20of%20the%20ARM%20templates%20is%20using%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fvirtual-machines%2Flinux%2Fusing-cloud-init%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ecloud%20init%3C%2FA%3E%20to%20configure%20the%20VMSS%20instances%20as%20they%20are%20created.%26nbsp%3B%20Below%20is%20the%20Unbuntu%20cloud%20init%20files.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH3%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%3ECloud%20Init%20for%20CEF-VMSS%3A%3C%2FH3%3E%0A%3CPRE%3E%23cloud-config%3CBR%20%2F%3Epackage_upgrade%3A%20true%3CBR%20%2F%3Eruncmd%3A%3CBR%20%2F%3E%26nbsp%3B%20-%20sudo%20apt-get%20update%3CBR%20%2F%3E%26nbsp%3B%20-%20sudo%20wget%20https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2FAzure-Sentinel%2Fmaster%2FDataConnectors%2FCEF%2Fcef_installer.py%26amp%3B%26amp%3Bsudo%20python%20cef_installer.py%3C%2FPRE%3E%0A%3CP%3EAs%20you%20can%20see%20the%20cloud%20init%2C%20it%20installs%20updates%20and%20the%20Log%20Analytics%20Agent%20using%20the%20Azure%20Sentinel%20CEF%20script.%26nbsp%3B%20The%20ARM%20template%20appends%20the%20workspace%20id%20and%20workspace%20key%20to%20the%20last%20line%20so%20that%20the%20agent%20gets%20connected%20to%20the%20right%20workspace%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH3%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%3ECloud%20Init%20for%20Logstash-VMSS%3A%3C%2FH3%3E%0A%3CPRE%3E%23cloud-config%3CBR%20%2F%3Epackage_upgrade%3A%20true%3CBR%20%2F%3Epackages%3A%3CBR%20%2F%3E%26nbsp%3B%20-%20default-jre%3CBR%20%2F%3Eruncmd%3A%3CBR%20%2F%3E%26nbsp%3B%20-%20wget%20-qO%20-%20https%3A%2F%2Fartifacts.elastic.co%2FGPG-KEY-elasticsearch%20%7C%20sudo%20apt-key%20add%20-%3CBR%20%2F%3E%26nbsp%3B%20-%20sudo%20apt-get%20install%20apt-transport-https%3CBR%20%2F%3E%26nbsp%3B%20-%20echo%20%22deb%20https%3A%2F%2Fartifacts.elastic.co%2Fpackages%2F7.x%2Fapt%20stable%20main%22%20%7C%20sudo%20tee%20-a%20%2Fetc%2Fapt%2Fsources.list.d%2Felastic-7.x.list%3CBR%20%2F%3E%26nbsp%3B%20-%20sudo%20apt-get%20update%3CBR%20%2F%3E%26nbsp%3B%20-%20sudo%20apt-get%20install%20logstash%3CBR%20%2F%3E%26nbsp%3B%20-%20sudo%20%2Fusr%2Fshare%2Flogstash%2Fbin%2Flogstash-plugin%20install%20logstash-output-syslog%3CBR%20%2F%3E%26nbsp%3B%20-%20sudo%20%2Fusr%2Fshare%2Flogstash%2Fbin%2Flogstash-plugin%20update%3CBR%20%2F%3E%26nbsp%3B%20-%20wget%20-q%20https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2FAzure-Sentinel%2Fmaster%2FDataConnectors%2FLogstash-VMSS%2Flogstash.config%20-O%20%2Fetc%2Flogstash%2Fconfig.d%2Flogstash.config%3CBR%20%2F%3E%26nbsp%3B%20-%20echo%20%22update%20this%20line%20with%20wget%20-q%20https%3A%2F%2FsourceURL%20-O%20%2Fetc%2Flogstash%2Fpipelines.yml%20if%20you%20have%20a%20custom%20pipelines%20file%22%3CBR%20%2F%3E%26nbsp%3B%20-%20sudo%20systemctl%20start%20logstash.service%3CBR%20%2F%3E%26nbsp%3B%20-%20sudo%20wget%20https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2FAzure-Sentinel%2Fmaster%2FDataConnectors%2FCEF%2Fcef_installer.py%26amp%3B%26amp%3Bsudo%20python%20cef_installer.py%3C%2FPRE%3E%0A%3CP%3EIt%20installs%20Java%2C%20Logstash%2C%20Logstash%20Syslog%20Output%20plugin%20and%20the%20Log%20Analytics%20Agent%20using%20the%20Azure%20Sentinel%20CEF%20script.%26nbsp%3B%20The%20ARM%20template%20appends%20the%20workspace%20id%20and%20workspace%20key%20to%20the%20last%20line%20so%20that%20the%20agent%20gets%20connected%20to%20the%20right%20workspace.%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%3ECEF%3C%2FH2%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fsentinel%2Fconnect-common-event-format%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3ECEF%3C%2FA%3E%20is%20our%20default%20way%20to%20collect%20external%20solutions%20like%20firewalls%20and%20proxies.%26nbsp%3B%20The%20CEF%20install%20script%20will%20install%20the%20Log%20Analytics%20agent%2C%20configure%20rsyslog%2C%20and%20configure%20the%20agent%20for%20CEF%20collection.%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%3ELogstash%3C%2FH2%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Findex.html%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3ELogstash%3C%2FA%3E%20dynamically%20ingests%2C%20transforms%2C%20and%20ships%20your%20data%20regardless%20of%20format%20or%20complexity.%26nbsp%3B%20It%20has%20many%20input%2C%20filter%20and%20output%20plugins.%26nbsp%3B%20This%20can%20allow%20you%20to%20get%20data%20from%20many%20sources%2C%20manipulate%20the%20event%20data%20and%20output%20to%20the%20Log%20Analytics%20Agent%20locally%20on%20the%20machine.%26nbsp%3B%20There%20are%20many%20input%20plugins%20so%20this%20makes%20it%20easy%20to%20connect%20to%20other%20sources%20like%20Kafka.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EHere%20is%20the%20sample%20logstash.conf%20file%20that%20is%20used%20in%20the%20sample%20architecture%3A%3C%2FP%3E%0A%3CPRE%3Einput%26nbsp%3B%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Btcp%26nbsp%3B%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bport%26nbsp%3B%3D%26gt%3B%26nbsp%3B5514%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Btype%26nbsp%3B%3D%26gt%3B%26nbsp%3Bsyslog%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bcodec%26nbsp%3B%3D%26gt%3B%26nbsp%3Bcef%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%7D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Budp%26nbsp%3B%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bport%26nbsp%3B%3D%26gt%3B%26nbsp%3B5514%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Btype%26nbsp%3B%3D%26gt%3B%26nbsp%3Bsyslog%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bcodec%26nbsp%3B%3D%26gt%3B%26nbsp%3Bcef%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%7D%3CBR%20%2F%3E%7D%3CBR%20%2F%3E%3CBR%20%2F%3Efilter%26nbsp%3B%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Bgeoip%26nbsp%3B%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bsource%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22src%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Btarget%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22srcGeoIP%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Badd_field%26nbsp%3B%3D%26gt%3B%26nbsp%3B%7B%26nbsp%3B%22sourceLongitude%22%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7B%5BsrcGeoIP%5D%5Blongitude%5D%7D%22%26nbsp%3B%7D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Badd_field%26nbsp%3B%3D%26gt%3B%26nbsp%3B%7B%26nbsp%3B%22sourceLatitude%22%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7B%5BsrcGeoIP%5D%5Blatitude%5D%7D%22%26nbsp%3B%7D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%7D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Bgeoip%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bsource%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22dst%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Btarget%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22dstGeoIP%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Badd_field%26nbsp%3B%3D%26gt%3B%26nbsp%3B%7B%26nbsp%3B%22destinationLongitude%22%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7B%5BdstGeoIP%5D%5Blongitude%5D%7D%22%26nbsp%3B%7D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Badd_field%26nbsp%3B%3D%26gt%3B%26nbsp%3B%7B%26nbsp%3B%22destinationLatitude%22%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7B%5BdstGeoIP%5D%5Blatitude%5D%7D%22%26nbsp%3B%7D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%7D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Bmutate%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Badd_field%26nbsp%3B%3D%26gt%3B%26nbsp%3B%7B%26nbsp%3B%22agentReceiptTime%22%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7B%40timestamp%7D%22%26nbsp%3B%7D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%7D%3CBR%20%2F%3E%7D%3CBR%20%2F%3E%3CBR%20%2F%3Eoutput%26nbsp%3B%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Bsyslog%26nbsp%3B%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Bhost%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22127.0.0.1%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Bport%26nbsp%3B%3D%26gt%3B%26nbsp%3B25226%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Bprotocol%26nbsp%3B%22tcp%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Bcodec%26nbsp%3B%3D%26gt%3B%26nbsp%3Bcef%26nbsp%3B%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Breverse_mapping%26nbsp%3B%3D%26gt%3B%26nbsp%3Btrue%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bdelimiter%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%5Cr%5Cn%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bvendor%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7BdeviceVendor%7D%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bproduct%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7BdeviceProduct%7D%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bversion%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7BdeviceVersion%7D%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bsignature%26nbsp%3B%26nbsp%3B%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7BdeviceEventClassId%7D%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bname%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7Bname%7D%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bseverity%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7Bseverity%7D%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bfields%26nbsp%3B%3D%26gt%3B%26nbsp%3B%5B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceAction%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22applicationProtocol%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomIPv6Address1%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomIPv6Address1Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomIPv6Address2%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomIPv6Address2Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomIPv6Address3%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomIPv6Address3Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomIPv6Address4%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomIPv6Address4Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceEventCategory%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomFloatingPoint1%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomFloatingPoint1Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomFloatingPoint2%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomFloatingPoint2Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomFloatingPoint3%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomFloatingPoint3Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomFloatingPoint4%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomFloatingPoint4Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomNumber1%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomNumber1Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomNumber2%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomNumber2Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomNumber3%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomNumber3Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22baseEventCount%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString1%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString1Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString2%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString2Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString3%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString3Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString4%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString4Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString5%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString5Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString6%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString6Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationHostName%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationMacAddress%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationNtDomain%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationProcessId%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationUserPrivileges%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationProcessName%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationPort%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationAddress%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationUserId%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationUserName%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceAddress%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceHostName%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceProcessId%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22endTime%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22fileName%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22fileSize%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22bytesIn%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22bytesOut%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22eventOutcome%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22transportProtocol%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22requestUrl%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceReceiptTime%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceHostName%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceMacAddress%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceNtDomain%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceProcessId%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceUserPrivileges%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceProcessName%22%2C%3CBR%20%2F%3E%20%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourcePort%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceAddress%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22startTime%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceUserId%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceUserName%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22agentHostName%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22agentReceiptTime%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22agentType%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22agentId%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22cefVersion%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22agentAddress%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22agentVersion%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22agentTimeZone%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationTimeZone%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceLongitude%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceLatitude%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationLongitude%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationLatitude%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22categoryDeviceType%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22managerReceiptTime%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22agentMacAddress%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%5D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%7D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%7D%3CBR%20%2F%3E%7D%3C%2FPRE%3E%0A%3CP%3EThe%20inputs%20accept%20both%20%3CA%20href%3D%22https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Fplugins-inputs-tcp.html%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3ETCP%3C%2FA%3E%20and%20%3CA%20href%3D%22https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Fplugins-inputs-udp.html%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EUDP%3C%2FA%3E%20on%20port%205514.%26nbsp%3B%20I%20used%205514%20because%20Logstash%20runs%20as%20non-root%20and%20requires%20special%20configuration%20to%20use%20port%20514.%26nbsp%3B%20I%20decided%20to%20keep%20it%20simple.%26nbsp%3B%20On%20input%2C%20its%20expecting%20%3CA%20href%3D%22https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Fplugins-codecs-cef.html%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3ECEF%3C%2FA%3E%20format%20using%20%E2%80%9Ccodec%20%3D%26gt%3B%20cef%E2%80%9D%20and%20tags%20the%20event%20as%20syslog.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOnce%20the%20event%20is%20accepted%2C%20I%20have%20added%20a%20few%20filters.%26nbsp%3B%20The%20first%20uses%20the%20%3CA%20href%3D%22https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Fplugins-filters-geoip.html%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EGeoIP%3C%2FA%3E%20plugin%20which%20uses%20the%20local%20GeoLite2%20database%20to%20lookup%20the%20source%20and%20destination%20IP%20addresses.%26nbsp%3B%20These%20are%20added%20to%20a%20custom%20field%20and%20to%20align%20with%20RFC%20compliance%20I%20then%20add_field%20to%20bring%20the%20latitude%20and%20longitude%20into%20proper%20fields.%26nbsp%3B%20I%20also%20%3CA%20href%3D%22https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Fplugins-filters-mutate.html%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3Emutate%3C%2FA%3E%20to%20copy%20the%20message%20received%20time%20into%20agentRecievedTime.%26nbsp%3B%20This%20is%20important%20as%20Logstash%20will%20send%20the%20message%20to%20the%20Log%20Analytics%20using%20its%20time%20which%20will%20end%20up%20as%20TimeGenerated%20in%20Log%20Analytics.%26nbsp%3B%20Doing%20this%20will%20allow%20you%20to%20see%20the%20original%20send%20time%20and%20the%20time%20Logstash%20sent%20it.%26nbsp%3B%20A%20simple%20compare%20will%20show%20you%20how%20long%20its%20take%20to%20process.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20the%20output%20section%2C%20I%20use%20the%20%3CA%20href%3D%22https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Fplugins-outputs-syslog.html%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3Esyslog%3C%2FA%3E%20plugin%20to%20output%20the%20message%20to%20the%20agent.%26nbsp%3B%20The%20agent%20listens%20on%20TCP%20127.0.0.1%3A25226.%26nbsp%3B%20I%20set%20the%20output%20plugin%20to%20the%20CEF%20codec%20again%20and%20there%20are%20couple%20of%20key%20important%20configs.%26nbsp%3B%20%E2%80%9Creverse_mapping%20%3D%26gt%3B%20true%E2%80%9D%20ensures%20that%20the%20message%20is%20sent%20using%20the%20short%20names%20(src%20vs%20sourceAddress)%20which%20is%20required%20by%20Log%20Analytics.%26nbsp%3B%20The%20fields%20portion%20requires%20all%20fields%20you%20want%20to%20send.%20I%20have%20included%20all%20fields%20the%20CEF%20codec%20supports.%26nbsp%3B%20If%20the%20fields%20doesn%E2%80%99t%20exist%20it%20wont%20be%20sent.%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%3EAzure%20Sentinel%3C%2FH2%3E%0A%3CP%3EOnce%20the%20data%20is%20sent%20to%20the%20agent%2C%20it%20will%20follow%20all%20the%20normal%20CEF%20collection%20ingestion%20process%20and%20end%20up%20in%20CommonSecurityLog.%26nbsp%3B%20You%20can%20monitor%20the%20VMSS%20event%20per%20second%20using%20the%20following%20query%3A%3C%2FP%3E%0A%3CPRE%3ECommonSecurityLog%3CBR%20%2F%3E%7C%20where%20_TimeReceived%20%26gt%3B%20ago(20m)%3CBR%20%2F%3E%7C%20summarize%20count()%20by%20bin(_TimeReceived%2C%201m)%2C%20_ResourceId%3CBR%20%2F%3E%7C%20extend%20count%20%3Dcount_%20%2F60%3CBR%20%2F%3E%7C%20sort%20by%20_TimeReceived%20desc%3C%2FPRE%3E%0A%3CP%3EThis%20will%20get%20all%20logs%20in%20the%20last%2020m%20and%20summarize%20by%20TimeRecieved%20and%20ResourceId.%26nbsp%3B%20This%20gives%20you%20the%20%23%20of%20event%20per%20minute%2C%20so%20you%20need%20to%20create%20a%20count%20column%20equal%20to%20count_%20divided%20by%2060%20seconds.%26nbsp%3B%20Now%20you%20can%20see%20the%20EPS%20per%20VMSS%20instance.%3C%2FP%3E%0A%3CP%3EIf%20you%20have%20performance%20issues%2C%20I%20recommend%20you%20look%20into%20Ryslog%20%3CA%20href%3D%22https%3A%2F%2Frsyslog.readthedocs.io%2Fen%2Flatest%2Fexamples%2Fhigh_performance.html%3Fhighlight%3Dperformance%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3Eperformance%3C%2FA%3E%20or%20Logstash%20%3CA%20href%3D%22https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Fperformance-tuning.html%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3Eperformance%20tuning%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ESome%20future%20improvements%20I%20might%20add%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EImplement%20impstats%20for%20rsyslog%20and%20send%20data%20to%20Log%20Analytics.%26nbsp%3B%20This%20would%20allow%20performance%20monitoring%20of%20rsyslog%20(dashboarding%2C%20queries)%3C%2FLI%3E%0A%3CLI%3EImplement%20GeoIP%20in%20rsyslog%3C%2FLI%3E%0A%3CLI%3EImplement%20Logstash%20monitoring%20APIs%20and%20send%20data%20to%20Log%20Analytics.%26nbsp%3B%20This%20would%20allow%20performance%20monitoring%20of%20Logstash%20(dashboarding%2C%20queries)%3C%2FLI%3E%0A%3CLI%3ECreate%20additional%20sample%20using%20fluentd%3C%2FLI%3E%0A%3CLI%3ECreate%20additional%20sample%20using%20syslog-ng%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EThanks%20for%20reading!%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1185854%22%20slang%3D%22en-US%22%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22CEF1.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F172375iEC8DB274FBECB6FB%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22CEF1.png%22%20alt%3D%22CEF1.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3ELearn%20how%20to%20scale%20up%20Syslog%20CEF%20collection%20in%20Azure%20Sentinel%20using%20Azure%20Virtual%20Machine%20Scale%20Sets%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1186188%22%20slang%3D%22en-US%22%3ERe%3A%20Scaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1186188%22%20slang%3D%22en-US%22%3E%3CP%3EI%20moved%20this%20feedback%20to%20the%20github%20repo%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1189072%22%20slang%3D%22en-US%22%3ERe%3A%20Scaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1189072%22%20slang%3D%22en-US%22%3EHi%20Nicholas%2C%20I%20was%20wondering%20if%20you%20know%20how%20I%20can%20daisy%20chain%20collectors%20like%20in%20the%20timed%20youtube%20link%20(%3CA%20href%3D%22https%3A%2F%2Fyoutu.be%2F_mm3GNwPBHU%3Flist%3DPLOhMGpMOPKRHPHCvzia3EE5OY5EkQRCuH%26amp%3Bt%3D896%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fyoutu.be%2F_mm3GNwPBHU%3Flist%3DPLOhMGpMOPKRHPHCvzia3EE5OY5EkQRCuH%26amp%3Bt%3D896%3C%2FA%3E)%20was%20presented%20by%20one%20of%20Microsoft's%20Sentinel%20guys%20called%20Ofer.%20I%20was%20able%20to%20install%20the%20CEF-Syslog%20Ubuntu%20server%20on-prem%20but%20I%20am%20trying%20to%20do%20the%20Syslog-Collector%20proxy.%20I%20believe%20there%20must%20be%20some%20configs%20that%20need%20to%20change%20on%20both%20the%20nodes.%20Simply%20the%20architecture%20I%20was%20to%20put%20is%20such%3A%201.%20Install%20a%20CEF-Syslog%20on-prem%20Ubuntu%20machine%20to%20collect%20all%20CEF%20and%20Syslog%20sources%202.%20Forward%20the%20data%20from%20the%20CEF-Syslog%20machine%20to%20another%20Syslog-Collector-Proxy%20in%20another%20segment%20of%20my%20network%20to%20forward%20to%20Azure%20to%20Sentinel%20Can%20you%20help%3F%20Thank%20you%2C%20Egal%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1535529%22%20slang%3D%22en-US%22%3ERe%3A%20Scaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1535529%22%20slang%3D%22en-US%22%3E%3CP%3EI%20see%26nbsp%3B%3CSPAN%3EStandard_B16ms%20sku%20is%20used%20in%20the%20vmss.%3C%2FSPAN%3E%3CBR%20%2F%3E%3C%2FP%3E%3CP%3E%3CSPAN%3EWas%20this%20a%20decision%20or%20just%20by%20sample%3F%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1536111%22%20slang%3D%22en-US%22%3ERe%3A%20Scaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1536111%22%20slang%3D%22en-US%22%3E%3CP%3EHi%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F602554%22%20target%3D%22_blank%22%3E%40eciruam%3C%2FA%3E%26nbsp%3BI%20actually%20need%20to%20update%20the%20template.%26nbsp%3B%20I%20worked%20with%20the%20agent%20team%20and%20they%20did%20some%20testing.%26nbsp%3B%20With%20an%26nbsp%3B%3CSTRONG%3EF4s_v2%20%3C%2FSTRONG%3EVM%20size%20they%20were%20able%20to%20achieve%20higher%20EPS%20and%20recommended%20that%20as%20the%20size.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1785297%22%20slang%3D%22en-US%22%3ERe%3A%20Scaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1785297%22%20slang%3D%22en-US%22%3E%3CP%3ENicholas%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Edo%20you%20have%20any%20concerns%20about%20scaling%20down%20a%20vm%20scale%20set%20without%20knowing%20if%20it's%20queue%20is%20empty%3F%26nbsp%3B%20isn't%20there%20some%20risk%20of%20lost%20messages%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20am%20toying%20with%20the%20idea%20of%20adapting%20my%20syslog-ng%20front-ends%20to%20push%20messages%20into%20a%20service%20bus%20queue%20(or%20multiple%20queues)%2C%20and%20then%20using%20a%20custom%20input%20into%20logstash%20to%20peek%20a%20message%20from%20the%20queue%2C%20process%20it%20through%20logstash%2C%20and%20then%20deque%20the%20message%20after%20processing.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThat%20would%20remove%20the%20need%20for%20load%20balancers%2C%20and%20allow%20the%20scale%20sets%20to%20expand%2Fcontract%20based%20on%20load%20(or%20ideally%20service%20bus%20queue%20sizes)%2C%20and%20guarantee%20delivery%20of%20the%20messages.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Ethoughts%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1785678%22%20slang%3D%22en-US%22%3ERe%3A%20Scaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1785678%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F11818%22%20target%3D%22_blank%22%3E%40Justin%20Ainsworth%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3Etheoretically%2C%20yea%20i%20guess%20that%20could%20happen.%26nbsp%3B%20but%20you%20could%20set%20autoscale%20settings%20to%20check%20network%20out%20instead%20of%20CPU.%26nbsp%3B%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fvirtual-machine-scale-sets%2Fvirtual-machine-scale-sets-autoscale-overview%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fvirtual-machine-scale-sets%2Fvirtual-machine-scale-sets-autoscale-overview%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3Eyea%20you%20could%20do%20some%20kind%20of%20bus%20in%20azure.%26nbsp%3B%20i%20went%20the%20simple%20route%20to%20solve%20a%20customer%20request.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2045392%22%20slang%3D%22en-US%22%3ERe%3A%20Scaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2045392%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20Nicolas%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIf%20someone's%20organisation%20policy%20does%20not%20allow%20SSH%20and%20public%20IP%20on%20load%20balancer%2C%20what%20is%20work%20around%20for%20this%20situation.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2119777%22%20slang%3D%22en-US%22%3ERe%3A%20Scaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2119777%22%20slang%3D%22en-US%22%3E%3CP%3EHi%26nbsp%3BNicholas%20DiCola%20(SECURITY%20JEDI)%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20have%20deployed%20the%20solution%20using%20the%20template%2C%20thank%20you%20for%20putting%20the%20work%20in%20to%20make%20it%20happen.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20am%20using%20the%20Ubuntu%20script.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20note%20when%20a%20VM%20is%20created%20the%20syslog%20service%20is%20not%20listening.%26nbsp%3B%20If%20I%20execute%20the%20cef_installer.py%20with%20workstation%20id%20and%20password%20from%20the%20root%20folder%20it%20reinstalls%2Freconfigures%20and%20syslog%20service%20is%20listening.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20am%20doing%20nothing%20different%20to%20the%20installer%20but%20getting%20a%20different%20outcome.%26nbsp%3B%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ECan%20you%20suggest%20anything%20to%20allow%20this%20to%20not%20require%20the%20human%20intervention%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThank%20you%2C%3C%2FP%3E%3CP%3EGeorge%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2149210%22%20slang%3D%22en-US%22%3ERe%3A%20Scaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2149210%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20Nicholas%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EMy%20Question%20might%20be%20somewhat%20in%20general.%3C%2FP%3E%3CP%3E1.)%20What%20types%20of%20data%20formats%20(Syslog%2C%20CEF%2C%20custom)%20can%20be%20taken%20up%20by%20Logstash%20for%20which%20it%20has%20some%20kind%20of%20mapping%20mechanism%20to%20ingest%20the%20data%20into%20Log%20Analytics%20workspace.%3C%2FP%3E%3CP%3E2.)%20Does%20Logstash%20have%20any%20mapping%20mechanism%20to%20map%20data%20format%20from%20any%20data%20source%20to%20convert%20into%20CEF%20or%20syslog%20which%20I%20suppose%20are%20the%20preferred%20choices%20of%20Sentinel%3F%3C%2FP%3E%3CP%3E3.)%20Do%20we%20really%20need%20Log%20Analytics%20Agent%20in%20between%20Logstash%20and%20Log%20Analytics%20workspace%20of%20sentinel%3F%3C%2FP%3E%3CP%3E4.)%20Which%20would%20be%20the%20best%20choice%20of%20data%20format%20(syslog%2C%20CEF%2C%20or%20custom)%20out%20from%20logstash%20to%20ingest%20into%20Log%20Analytics%20workspace%20of%20sentinel%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E5.)%20Can%20the%20logstash-vmss%20be%20deployed%20on-prem%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E6.)%20Could%20you%20please%20suggest%20to%20me%20the%20best%20choices%20of%20data%20intake%20format%20(I%20do%20understand%20various%20data%20sources%20may%20have%20their%20own%20data%20formats%20)%20to%20Logstash%20from%20any%20data%20sources%20and%20data%20output%20format%20from%20Logstash%20to%20Log%20Analytics%20workspace%20of%20sentinel%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ERegards%2C%3C%2FP%3E%3CP%3ESimranjeet%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2149735%22%20slang%3D%22en-US%22%3ERe%3A%20Scaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2149735%22%20slang%3D%22en-US%22%3E%3CP%20class%3D%22p1%22%3E1.)%20What%20types%20of%20data%20formats%20(Syslog%2C%20CEF%2C%20custom)%20can%20be%20taken%20up%20by%20Logstash%20for%20which%20it%20has%20some%20kind%20of%20mapping%20mechanism%20to%20ingest%20the%20data%20into%20Log%20Analytics%20workspace.%3C%2FP%3E%0A%3CP%20class%3D%22p2%22%3E%3CSPAN%20class%3D%22s1%22%3ELogstash%20is%20an%20event%20pipeline%20system.%3CSPAN%20class%3D%22Apple-converted-space%22%3E%26nbsp%3B%20%3C%2FSPAN%3Eit%20has%20many%20input%20plugins%20%3CA%20href%3D%22https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Finput-plugins.html%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Finput-plugins.html%3C%2FA%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%20class%3D%22p1%22%3EIt%20can%20also%20transform%20data%20during%20parsing.%3CSPAN%20class%3D%22Apple-converted-space%22%3E%26nbsp%3B%20%3C%2FSPAN%3ESo%20to%20answer%20many%20data%20formats%20can%20be%20changed%20to%20match%20CEF%20output.%3CSPAN%20class%3D%22Apple-converted-space%22%3E%26nbsp%3B%20%3C%2FSPAN%3EThis%20VMSS%20solution%20was%20designed%20specific%20to%20solve%20customer%20need%20to%20get%20SYSLOG%20(CEF%20FORMAT)%20messages%2C%20add%20geo%20ip%20information%2C%20then%20send%20to%20log%20analytics.%3C%2FP%3E%0A%3CP%20class%3D%22p1%22%3E2.)%20Does%20Logstash%20have%20any%20mapping%20mechanism%20to%20map%20data%20format%20from%20any%20data%20source%20to%20convert%20into%20CEF%20or%20syslog%20which%20I%20suppose%20are%20the%20preferred%20choices%20of%20Sentinel%3F%3C%2FP%3E%0A%3CP%20class%3D%22p1%22%3EYes%20please%20review%20on%20Logstash%20documentation.%3C%2FP%3E%0A%3CP%20class%3D%22p1%22%3E3.)%20Do%20we%20really%20need%20Log%20Analytics%20Agent%20in%20between%20Logstash%20and%20Log%20Analytics%20workspace%20of%20sentinel%3F%3C%2FP%3E%0A%3CP%20class%3D%22p1%22%3EFor%20this%20scenario%2C%20yes.%3CSPAN%20class%3D%22Apple-converted-space%22%3E%26nbsp%3B%20%3C%2FSPAN%3ECommonEventLogs%20come%20from%20the%20agent%20only.%3CSPAN%20class%3D%22Apple-converted-space%22%3E%26nbsp%3B%20%3C%2FSPAN%3EYou%20could%20change%20the%20output%20to%20Custom%20Logs%20using%20the%20output%20connector%20for%20Log%20Analytics.%3C%2FP%3E%0A%3CP%20class%3D%22p1%22%3E4.)%20Which%20would%20be%20the%20best%20choice%20of%20data%20format%20(syslog%2C%20CEF%2C%20or%20custom)%20out%20from%20logstash%20to%20ingest%20into%20Log%20Analytics%20workspace%20of%20sentinel%3F%3C%2FP%3E%0A%3CP%20class%3D%22p1%22%3EThere%20is%20no%20simple%20answer.%3CSPAN%20class%3D%22Apple-converted-space%22%3E%26nbsp%3B%20%3C%2FSPAN%3EDepends%20on%20the%20data%20source.%3C%2FP%3E%0A%3CP%20class%3D%22p1%22%3E%26nbsp%3B5.)%20Can%20the%20logstash-vmss%20be%20deployed%20on-prem%3F%3C%2FP%3E%0A%3CP%20class%3D%22p1%22%3ELogstash%20can%20be%20deployed%20anywhere.%3CSPAN%20class%3D%22Apple-converted-space%22%3E%26nbsp%3B%20%3C%2FSPAN%3ESo%20can%20the%20Log%20A%20Agent.%3CSPAN%20class%3D%22Apple-converted-space%22%3E%26nbsp%3B%20%3C%2FSPAN%3EThe%20VMSS%20is%20specific%20to%20Azure%20resource.%3CSPAN%20class%3D%22Apple-converted-space%22%3E%26nbsp%3B%20%3C%2FSPAN%3EYou%20would%20need%20to%20write%20your%20own%20installer%20scripts%20for%20on-prem%3C%2FP%3E%0A%3CP%20class%3D%22p1%22%3E%26nbsp%3B6.)%20Could%20you%20please%20suggest%20to%20me%20the%20best%20choices%20of%20data%20intake%20format%20(I%20do%20understand%20various%20data%20sources%20may%20have%20their%20own%20data%20formats%20)%20to%20Logstash%20from%20any%20data%20sources%20and%20data%20output%20format%20from%20Logstash%20to%20Log%20Analytics%20workspace%20of%20sentinel%3F%3C%2FP%3E%0A%3CP%20class%3D%22p1%22%3EIt%20depends%20on%20your%20data%20source%3F%3CSPAN%20class%3D%22Apple-converted-space%22%3E%26nbsp%3B%20%3C%2FSPAN%3EIs%20it%20a%20network%20Appliance%20like%20a%20firewall%2C%20normally%20that%E2%80%99s%20CEF.%3CSPAN%20class%3D%22Apple-converted-space%22%3E%26nbsp%3B%20%3C%2FSPAN%3EWindows%20Event%20Logs%20should%20go%20to%20windows%20event%20or%20security%20event.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2155395%22%20slang%3D%22en-US%22%3ERe%3A%20Scaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2155395%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20Nichola%3C%2FP%3E%3CP%3EI%20am%20grateful%20for%20the%20replies%20to%20my%20previous%20question.%3C%2FP%3E%3CP%3ESince%20you%20have%20told%20me%20that%20Logstash%20and%20Log%20A%20Agent%20can%20be%20deployed%20anywhere%20and%20I%20have%20seen%20on%20this%20page%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fsentinel%2Fconnect-logstash%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fsentinel%2Fconnect-logstash%3C%2FA%3E%26nbsp%3Bhow%20to%20configure%20Logstash%20and%20send%20logs%20Log%20Analytics%20workspace%20using%20a%20log%20A%20output%20plugin.%3C%2FP%3E%3CP%3EI%20wish%20to%20know%3A%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E1.)%20In%20the%20above%20link%2C%20there%20is%20no%20mention%20of%20Log%20A%20Agent%20which%20should%20be%20there%20in%20the%20case%20of%20Logstash-vmss's%20architecture.%20As%20per%20my%20understanding%2C%20the%20Log%20A%20output%20plugin%20takes%20logs%20from%20Logstash%20and%20ingests%20them%20into%20Log%20A%20workspace%20in%20a%20custom%20table%20which%20I%20suppose%20is%20neither%20syslog%20no%20cef%20format.%20Am%20I%20correct%20at%20this%20point%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E2.)%20Can%20we%20build%20such%20a%20model%20where%20the%20whole%20of%20Microsoft's%20grand%20list%20of%20data%20sources(%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fazure-sentinel%2Fazure-sentinel-the-connectors-grand-cef-syslog-direct-agent%2Fba-p%2F803891%22%20target%3D%22_blank%22%3Ehttps%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fazure-sentinel%2Fazure-sentinel-the-connectors-grand-cef-syslog-direct-agent%2Fba-p%2F803891%3C%2FA%3E)%26nbsp%3B%20be%20ingested%20into%20sentinel%20in%20one%20single%20common%20format%20(cef%2Fsyslog%20not%20custom)%20using%20Logstash%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E3.)%20I%20read%20somewhere%20sentinel%20gives%20better%20monitoring%2C%20analytics%2C%20correlation%2C%20incident%20generation%2C%20etc.%20if%20data%20from%20all%20data%20sources%20be%20ingested%20into%20sentinel%20in%20cef%2Fsyslog%20format.%20There%20can%20be%20quality%20issues%20if%20every%20data%20source%20has%20its%20own%20custom%20data%20format%20and%20table%20in%20sentinel%20which%20will%20not%20allow%20sentinel%20to%20do%20better%20analytics%20on%20data%20because%20of%20the%20randomness%20of%20data%20field%20names.%20Am%20I%20correct%20on%20this%20point%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E4.)%20Can%20sentinel%20perform%20data%20correlation%20and%20analytics%20if%20there%20are%20N%20numbers%20of%20custom%20tables%20present%20for%20different%20data%20source%20security%20appliances%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ENicholas%2C%20I%20am%20sorry%20if%20I%20am%20eating%20your%20mind%20too%20much.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ERegards%2C%3C%2FP%3E%3CP%3ESimran%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2155877%22%20slang%3D%22en-US%22%3ERe%3A%20Scaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2155877%22%20slang%3D%22en-US%22%3E%3CP%3E1.)%20In%20the%20above%20link%2C%20there%20is%20no%20mention%20of%20Log%20A%20Agent%20which%20should%20be%20there%20in%20the%20case%20of%20Logstash-vmss's%20architecture.%20As%20per%20my%20understanding%2C%20the%20Log%20A%20output%20plugin%20takes%20logs%20from%20Logstash%20and%20ingests%20them%20into%20Log%20A%20workspace%20in%20a%20custom%20table%20which%20I%20suppose%20is%20neither%20syslog%20no%20cef%20format.%20Am%20I%20correct%20at%20this%20point%3F%3C%2FP%3E%0A%3CP%3Ethe%20document%20is%20how%20to%20connect%20logstash%20to%20send%20custom%20logs.%26nbsp%3B%20The%20VMSS%20is%20to%20send%20syslog%20to%20CEF%20table%20in%20log%20a.%26nbsp%3B%20you%20can%20use%20either%20option%2C%20but%20the%20VMSS%20was%20very%20specific%20to%20get%20CEF%20logs%20into%20CEF%20table.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E2.)%20Can%20we%20build%20such%20a%20model%20where%20the%20whole%20of%20Microsoft's%20grand%20list%20of%20data%20sources(%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fazure-sentinel%2Fazure-sentinel-the-connectors-grand-cef-syslog-direct-agent%2Fba-p%2F803891%22%20target%3D%22_blank%22%3Ehttps%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fazure-sentinel%2Fazure-sentinel-the-connectors-grand-cef-syslog...%3C%2FA%3E)%26nbsp%3B%20be%20ingested%20into%20sentinel%20in%20one%20single%20common%20format%20(cef%2Fsyslog%20not%20custom)%20using%20Logstash%3F%3C%2FP%3E%0A%3CP%3EYes%20you%20could.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E3.)%20I%20read%20somewhere%20sentinel%20gives%20better%20monitoring%2C%20analytics%2C%20correlation%2C%20incident%20generation%2C%20etc.%20if%20data%20from%20all%20data%20sources%20be%20ingested%20into%20sentinel%20in%20cef%2Fsyslog%20format.%20There%20can%20be%20quality%20issues%20if%20every%20data%20source%20has%20its%20own%20custom%20data%20format%20and%20table%20in%20sentinel%20which%20will%20not%20allow%20sentinel%20to%20do%20better%20analytics%20on%20data%20because%20of%20the%20randomness%20of%20data%20field%20names.%20Am%20I%20correct%20on%20this%20point%3F%3C%2FP%3E%0A%3CP%3ECorrect.%26nbsp%3B%20CEF%20is%20a%20standard%20format%20so%20running%20queries%20is%20much%20easier%20when%20all%20syslog%20data%20is%20in%20the%20same%20format.%26nbsp%3B%20if%20each%20has%20its%20own%20custom%20log%20then%20you%20need%20to%20write%20queries%20for%20each%20custom%20source.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E4.)%20Can%20sentinel%20perform%20data%20correlation%20and%20analytics%20if%20there%20are%20N%20numbers%20of%20custom%20tables%20present%20for%20different%20data%20source%20security%20appliances%3F%3C%2FP%3E%0A%3CP%3EYes%20but%20it%20requires%20a%20more%20complex%20query.%26nbsp%3B%20Hence%20using%20CEF%20makes%20it%20easier.%3C%2FP%3E%3C%2FLINGO-BODY%3E
Version history
Last update:
‎Feb 20 2020 04:13 PM