%3CLINGO-SUB%20id%3D%22lingo-sub-1185854%22%20slang%3D%22en-US%22%3EScaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1185854%22%20slang%3D%22en-US%22%3E%3CP%3E%3CEM%3EThis%20blog%20post%20is%20authored%20by%20%3C%2FEM%3E%3CA%20href%3D%22https%3A%2F%2Ftwitter.com%2FMasterSecJedi%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3E%3CEM%3ENicholas%20DiCola%3C%2FEM%3E%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20the%20last%20few%20months%20working%20on%20Azure%20Sentinel%2C%20many%20customers%20have%20asked%20me%20about%20scaling%20up%20syslog%20CEF%20collection%20for%20getting%20data%20into%20Azure%20Sentinel.%26nbsp%3B%20I%20have%20created%20two%20sample%20architectures%20with%20code%20deployment%20for%20this%20purpose.%26nbsp%3B%20The%20samples%20is%20available%20at%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FAzure%2FAzure-Sentinel%2Ftree%2Fmaster%2FDataConnectors%2FCEF-VMSS%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fgithub.com%2FAzure%2FAzure-Sentinel%2Ftree%2Fmaster%2FDataConnectors%2FCEF-VMSS%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FAzure%2FAzure-Sentinel%2Ftree%2Fmaster%2FDataConnectors%2FLogstash-VMSS%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fgithub.com%2FAzure%2FAzure-Sentinel%2Ftree%2Fmaster%2FDataConnectors%2FLogstash-VMSS%3C%2FA%3E%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ECEF-VMSS%20is%20for%20deploying%20native%20Azure%20Sentinel%20CEF%20collection%20by%20sending%20syslog%20CEF%20message%20to%20rsyslog%20which%20then%20sends%20the%20messages%20to%20the%20Log%20Analytics%20Agent.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ELogstash-VMSS%20is%20for%20deploying%20Logstash%20on%20the%20VMs%20to%20do%20message%20manipulation%20which%20then%20sends%20the%20messages%20to%20the%20Log%20Analytics%20Agent.%26nbsp%3B%20You%20may%20also%20want%20to%20use%20this%20architecture%20and%20change%20the%20input%20to%20a%20source%20like%20Kafka.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EI%20will%20not%20deep%20dive%20on%20all%20the%20topics%20of%20this%20architecture.%26nbsp%3B%20You%20can%20research%20each%20on%20your%20own%20and%20will%20focus%20on%20an%20overview%20of%20the%20architecture.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%20id%3D%22toc-hId--1408921410%22%3EVirtual%20Machine%20Scale%20Set%3C%2FH2%3E%0A%3CP%3EThe%20architecture%20starts%20with%20a%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fvirtual-machine-scale-sets%2Foverview%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EVMSS%3C%2FA%3E%20which%20lets%20you%20manage%20and%20create%20a%20group%20of%20virtual%20machines.%26nbsp%3B%20These%20VMs%20can%20autoscale%20up%20and%20down%20additional%20instances%20based%20on%20schedule%20or%20demand.%26nbsp%3B%20The%20sample%20uses%20autoscale%20settings%20to%20configure%20the%20VMSS%20to%20scale%20up%20and%20down%20based%20on%20CPU%20(demand)%20of%20messages%20being%20sent.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EI%20have%20included%20a%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fload-balancer%2Fload-balancer-overview%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3ELoad%20Balancer%3C%2FA%3E%20in-front%20of%20the%20VMSS%20which%20will%20allow%20you%20to%20configure%201%20destination%20IP%20address%20(the%20Public%20Ip%20Address)%20and%20it%20will%20spread%20the%20incoming%20messages%20across%20the%20running%20instances.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThere%20are%202%20ARM%20templates%20for%20RedHat%20and%20Unbuntu.%26nbsp%3B%20The%20templates%20deploy%20everything%20needed%20for%20the%20architecture.%26nbsp%3B%20One%20key%20part%20of%20the%20ARM%20templates%20is%20using%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fvirtual-machines%2Flinux%2Fusing-cloud-init%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ecloud%20init%3C%2FA%3E%20to%20configure%20the%20VMSS%20instances%20as%20they%20are%20created.%26nbsp%3B%20Below%20is%20the%20Unbuntu%20cloud%20init%20files.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH3%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%20id%3D%22toc-hId--718359936%22%3ECloud%20Init%20for%20CEF-VMSS%3A%3C%2FH3%3E%0A%3CPRE%3E%23cloud-config%3CBR%20%2F%3Epackage_upgrade%3A%20true%3CBR%20%2F%3Eruncmd%3A%3CBR%20%2F%3E%26nbsp%3B%20-%20sudo%20apt-get%20update%3CBR%20%2F%3E%26nbsp%3B%20-%20sudo%20wget%20https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2FAzure-Sentinel%2Fmaster%2FDataConnectors%2FCEF%2Fcef_installer.py%26amp%3B%26amp%3Bsudo%20python%20cef_installer.py%3C%2FPRE%3E%0A%3CP%3EAs%20you%20can%20see%20the%20cloud%20init%2C%20it%20installs%20updates%20and%20the%20Log%20Analytics%20Agent%20using%20the%20Azure%20Sentinel%20CEF%20script.%26nbsp%3B%20The%20ARM%20template%20appends%20the%20workspace%20id%20and%20workspace%20key%20to%20the%20last%20line%20so%20that%20the%20agent%20gets%20connected%20to%20the%20right%20workspace%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH3%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%20id%3D%22toc-hId-1769152897%22%3ECloud%20Init%20for%20Logstash-VMSS%3A%3C%2FH3%3E%0A%3CPRE%3E%23cloud-config%3CBR%20%2F%3Epackage_upgrade%3A%20true%3CBR%20%2F%3Epackages%3A%3CBR%20%2F%3E%26nbsp%3B%20-%20default-jre%3CBR%20%2F%3Eruncmd%3A%3CBR%20%2F%3E%26nbsp%3B%20-%20wget%20-qO%20-%20https%3A%2F%2Fartifacts.elastic.co%2FGPG-KEY-elasticsearch%20%7C%20sudo%20apt-key%20add%20-%3CBR%20%2F%3E%26nbsp%3B%20-%20sudo%20apt-get%20install%20apt-transport-https%3CBR%20%2F%3E%26nbsp%3B%20-%20echo%20%22deb%20https%3A%2F%2Fartifacts.elastic.co%2Fpackages%2F7.x%2Fapt%20stable%20main%22%20%7C%20sudo%20tee%20-a%20%2Fetc%2Fapt%2Fsources.list.d%2Felastic-7.x.list%3CBR%20%2F%3E%26nbsp%3B%20-%20sudo%20apt-get%20update%3CBR%20%2F%3E%26nbsp%3B%20-%20sudo%20apt-get%20install%20logstash%3CBR%20%2F%3E%26nbsp%3B%20-%20sudo%20%2Fusr%2Fshare%2Flogstash%2Fbin%2Flogstash-plugin%20install%20logstash-output-syslog%3CBR%20%2F%3E%26nbsp%3B%20-%20sudo%20%2Fusr%2Fshare%2Flogstash%2Fbin%2Flogstash-plugin%20update%3CBR%20%2F%3E%26nbsp%3B%20-%20wget%20-q%20https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2FAzure-Sentinel%2Fmaster%2FDataConnectors%2FLogstash-VMSS%2Flogstash.config%20-O%20%2Fetc%2Flogstash%2Fconfig.d%2Flogstash.config%3CBR%20%2F%3E%26nbsp%3B%20-%20echo%20%22update%20this%20line%20with%20wget%20-q%20https%3A%2F%2FsourceURL%20-O%20%2Fetc%2Flogstash%2Fpipelines.yml%20if%20you%20have%20a%20custom%20pipelines%20file%22%3CBR%20%2F%3E%26nbsp%3B%20-%20sudo%20systemctl%20start%20logstash.service%3CBR%20%2F%3E%26nbsp%3B%20-%20sudo%20wget%20https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2FAzure-Sentinel%2Fmaster%2FDataConnectors%2FCEF%2Fcef_installer.py%26amp%3B%26amp%3Bsudo%20python%20cef_installer.py%3C%2FPRE%3E%0A%3CP%3EIt%20installs%20Java%2C%20Logstash%2C%20Logstash%20Syslog%20Output%20plugin%20and%20the%20Log%20Analytics%20Agent%20using%20the%20Azure%20Sentinel%20CEF%20script.%26nbsp%3B%20The%20ARM%20template%20appends%20the%20workspace%20id%20and%20workspace%20key%20to%20the%20last%20line%20so%20that%20the%20agent%20gets%20connected%20to%20the%20right%20workspace.%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%20id%3D%22toc-hId-1758649793%22%3ECEF%3C%2FH2%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fsentinel%2Fconnect-common-event-format%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3ECEF%3C%2FA%3E%20is%20our%20default%20way%20to%20collect%20external%20solutions%20like%20firewalls%20and%20proxies.%26nbsp%3B%20The%20CEF%20install%20script%20will%20install%20the%20Log%20Analytics%20agent%2C%20configure%20rsyslog%2C%20and%20configure%20the%20agent%20for%20CEF%20collection.%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%20id%3D%22toc-hId--48804670%22%3ELogstash%3C%2FH2%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Findex.html%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3ELogstash%3C%2FA%3E%20dynamically%20ingests%2C%20transforms%2C%20and%20ships%20your%20data%20regardless%20of%20format%20or%20complexity.%26nbsp%3B%20It%20has%20many%20input%2C%20filter%20and%20output%20plugins.%26nbsp%3B%20This%20can%20allow%20you%20to%20get%20data%20from%20many%20sources%2C%20manipulate%20the%20event%20data%20and%20output%20to%20the%20Log%20Analytics%20Agent%20locally%20on%20the%20machine.%26nbsp%3B%20There%20are%20many%20input%20plugins%20so%20this%20makes%20it%20easy%20to%20connect%20to%20other%20sources%20like%20Kafka.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EHere%20is%20the%20sample%20logstash.conf%20file%20that%20is%20used%20in%20the%20sample%20architecture%3A%3C%2FP%3E%0A%3CPRE%3Einput%26nbsp%3B%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Btcp%26nbsp%3B%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bport%26nbsp%3B%3D%26gt%3B%26nbsp%3B5514%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Btype%26nbsp%3B%3D%26gt%3B%26nbsp%3Bsyslog%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bcodec%26nbsp%3B%3D%26gt%3B%26nbsp%3Bcef%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%7D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Budp%26nbsp%3B%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bport%26nbsp%3B%3D%26gt%3B%26nbsp%3B5514%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Btype%26nbsp%3B%3D%26gt%3B%26nbsp%3Bsyslog%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bcodec%26nbsp%3B%3D%26gt%3B%26nbsp%3Bcef%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%7D%3CBR%20%2F%3E%7D%3CBR%20%2F%3E%3CBR%20%2F%3Efilter%26nbsp%3B%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Bgeoip%26nbsp%3B%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bsource%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22src%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Btarget%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22srcGeoIP%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Badd_field%26nbsp%3B%3D%26gt%3B%26nbsp%3B%7B%26nbsp%3B%22sourceLongitude%22%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7B%5BsrcGeoIP%5D%5Blongitude%5D%7D%22%26nbsp%3B%7D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Badd_field%26nbsp%3B%3D%26gt%3B%26nbsp%3B%7B%26nbsp%3B%22sourceLatitude%22%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7B%5BsrcGeoIP%5D%5Blatitude%5D%7D%22%26nbsp%3B%7D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%7D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Bgeoip%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bsource%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22dst%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Btarget%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22dstGeoIP%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Badd_field%26nbsp%3B%3D%26gt%3B%26nbsp%3B%7B%26nbsp%3B%22destinationLongitude%22%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7B%5BdstGeoIP%5D%5Blongitude%5D%7D%22%26nbsp%3B%7D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Badd_field%26nbsp%3B%3D%26gt%3B%26nbsp%3B%7B%26nbsp%3B%22destinationLatitude%22%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7B%5BdstGeoIP%5D%5Blatitude%5D%7D%22%26nbsp%3B%7D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%7D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Bmutate%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Badd_field%26nbsp%3B%3D%26gt%3B%26nbsp%3B%7B%26nbsp%3B%22agentReceiptTime%22%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7B%40timestamp%7D%22%26nbsp%3B%7D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%7D%3CBR%20%2F%3E%7D%3CBR%20%2F%3E%3CBR%20%2F%3Eoutput%26nbsp%3B%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Bsyslog%26nbsp%3B%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Bhost%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22127.0.0.1%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Bport%26nbsp%3B%3D%26gt%3B%26nbsp%3B25226%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Bprotocol%26nbsp%3B%22tcp%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3Bcodec%26nbsp%3B%3D%26gt%3B%26nbsp%3Bcef%26nbsp%3B%7B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Breverse_mapping%26nbsp%3B%3D%26gt%3B%26nbsp%3Btrue%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bdelimiter%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%5Cr%5Cn%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bvendor%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7BdeviceVendor%7D%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bproduct%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7BdeviceProduct%7D%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bversion%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7BdeviceVersion%7D%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bsignature%26nbsp%3B%26nbsp%3B%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7BdeviceEventClassId%7D%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bname%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7Bname%7D%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bseverity%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3D%26gt%3B%26nbsp%3B%22%25%7Bseverity%7D%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3Bfields%26nbsp%3B%3D%26gt%3B%26nbsp%3B%5B%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceAction%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22applicationProtocol%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomIPv6Address1%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomIPv6Address1Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomIPv6Address2%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomIPv6Address2Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomIPv6Address3%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomIPv6Address3Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomIPv6Address4%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomIPv6Address4Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceEventCategory%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomFloatingPoint1%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomFloatingPoint1Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomFloatingPoint2%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomFloatingPoint2Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomFloatingPoint3%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomFloatingPoint3Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomFloatingPoint4%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomFloatingPoint4Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomNumber1%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomNumber1Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomNumber2%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomNumber2Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomNumber3%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomNumber3Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22baseEventCount%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString1%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString1Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString2%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString2Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString3%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString3Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString4%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString4Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString5%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString5Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString6%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceCustomString6Label%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationHostName%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationMacAddress%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationNtDomain%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationProcessId%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationUserPrivileges%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationProcessName%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationPort%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationAddress%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationUserId%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationUserName%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceAddress%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceHostName%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceProcessId%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22endTime%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22fileName%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22fileSize%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22bytesIn%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22bytesOut%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22eventOutcome%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22transportProtocol%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22requestUrl%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22deviceReceiptTime%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceHostName%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceMacAddress%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceNtDomain%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceProcessId%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceUserPrivileges%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceProcessName%22%2C%3CBR%20%2F%3E%20%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourcePort%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceAddress%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22startTime%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceUserId%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceUserName%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22agentHostName%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22agentReceiptTime%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22agentType%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22agentId%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22cefVersion%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22agentAddress%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22agentVersion%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22agentTimeZone%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationTimeZone%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceLongitude%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22sourceLatitude%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationLongitude%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22destinationLatitude%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22categoryDeviceType%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22managerReceiptTime%22%2C%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%22agentMacAddress%22%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%5D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%7D%3CBR%20%2F%3E%26nbsp%3B%26nbsp%3B%7D%3CBR%20%2F%3E%7D%3C%2FPRE%3E%0A%3CP%3EThe%20inputs%20accept%20both%20%3CA%20href%3D%22https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Fplugins-inputs-tcp.html%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3ETCP%3C%2FA%3E%20and%20%3CA%20href%3D%22https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Fplugins-inputs-udp.html%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EUDP%3C%2FA%3E%20on%20port%205514.%26nbsp%3B%20I%20used%205514%20because%20Logstash%20runs%20as%20non-root%20and%20requires%20special%20configuration%20to%20use%20port%20514.%26nbsp%3B%20I%20decided%20to%20keep%20it%20simple.%26nbsp%3B%20On%20input%2C%20its%20expecting%20%3CA%20href%3D%22https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Fplugins-codecs-cef.html%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3ECEF%3C%2FA%3E%20format%20using%20%E2%80%9Ccodec%20%3D%26gt%3B%20cef%E2%80%9D%20and%20tags%20the%20event%20as%20syslog.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOnce%20the%20event%20is%20accepted%2C%20I%20have%20added%20a%20few%20filters.%26nbsp%3B%20The%20first%20uses%20the%20%3CA%20href%3D%22https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Fplugins-filters-geoip.html%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EGeoIP%3C%2FA%3E%20plugin%20which%20uses%20the%20local%20GeoLite2%20database%20to%20lookup%20the%20source%20and%20destination%20IP%20addresses.%26nbsp%3B%20These%20are%20added%20to%20a%20custom%20field%20and%20to%20align%20with%20RFC%20compliance%20I%20then%20add_field%20to%20bring%20the%20latitude%20and%20longitude%20into%20proper%20fields.%26nbsp%3B%20I%20also%20%3CA%20href%3D%22https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Fplugins-filters-mutate.html%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Emutate%3C%2FA%3E%20to%20copy%20the%20message%20received%20time%20into%20agentRecievedTime.%26nbsp%3B%20This%20is%20important%20as%20Logstash%20will%20send%20the%20message%20to%20the%20Log%20Analytics%20using%20its%20time%20which%20will%20end%20up%20as%20TimeGenerated%20in%20Log%20Analytics.%26nbsp%3B%20Doing%20this%20will%20allow%20you%20to%20see%20the%20original%20send%20time%20and%20the%20time%20Logstash%20sent%20it.%26nbsp%3B%20A%20simple%20compare%20will%20show%20you%20how%20long%20its%20take%20to%20process.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20the%20output%20section%2C%20I%20use%20the%20%3CA%20href%3D%22https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Fplugins-outputs-syslog.html%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Esyslog%3C%2FA%3E%20plugin%20to%20output%20the%20message%20to%20the%20agent.%26nbsp%3B%20The%20agent%20listens%20on%20TCP%20127.0.0.1%3A25226.%26nbsp%3B%20I%20set%20the%20output%20plugin%20to%20the%20CEF%20codec%20again%20and%20there%20are%20couple%20of%20key%20important%20configs.%26nbsp%3B%20%E2%80%9Creverse_mapping%20%3D%26gt%3B%20true%E2%80%9D%20ensures%20that%20the%20message%20is%20sent%20using%20the%20short%20names%20(src%20vs%20sourceAddress)%20which%20is%20required%20by%20Log%20Analytics.%26nbsp%3B%20The%20fields%20portion%20requires%20all%20fields%20you%20want%20to%20send.%20I%20have%20included%20all%20fields%20the%20CEF%20codec%20supports.%26nbsp%3B%20If%20the%20fields%20doesn%E2%80%99t%20exist%20it%20wont%20be%20sent.%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%20id%3D%22toc-hId--1856259133%22%3EAzure%20Sentinel%3C%2FH2%3E%0A%3CP%3EOnce%20the%20data%20is%20sent%20to%20the%20agent%2C%20it%20will%20follow%20all%20the%20normal%20CEF%20collection%20ingestion%20process%20and%20end%20up%20in%20CommonSecurityLog.%26nbsp%3B%20You%20can%20monitor%20the%20VMSS%20event%20per%20second%20using%20the%20following%20query%3A%3C%2FP%3E%0A%3CPRE%3ECommonSecurityLog%3CBR%20%2F%3E%7C%20where%20_TimeReceived%20%26gt%3B%20ago(20m)%3CBR%20%2F%3E%7C%20summarize%20count()%20by%20bin(_TimeReceived%2C%201m)%2C%20_ResourceId%3CBR%20%2F%3E%7C%20extend%20count%20%3Dcount_%20%2F60%3CBR%20%2F%3E%7C%20sort%20by%20_TimeReceived%20desc%3C%2FPRE%3E%0A%3CP%3EThis%20will%20get%20all%20logs%20in%20the%20last%2020m%20and%20summarize%20by%20TimeRecieved%20and%20ResourceId.%26nbsp%3B%20This%20gives%20you%20the%20%23%20of%20event%20per%20minute%2C%20so%20you%20need%20to%20create%20a%20count%20column%20equal%20to%20count_%20divided%20by%2060%20seconds.%26nbsp%3B%20Now%20you%20can%20see%20the%20EPS%20per%20VMSS%20instance.%3C%2FP%3E%0A%3CP%3EIf%20you%20have%20performance%20issues%2C%20I%20recommend%20you%20look%20into%20Ryslog%20%3CA%20href%3D%22https%3A%2F%2Frsyslog.readthedocs.io%2Fen%2Flatest%2Fexamples%2Fhigh_performance.html%3Fhighlight%3Dperformance%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Eperformance%3C%2FA%3E%20or%20Logstash%20%3CA%20href%3D%22https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Flogstash%2Fcurrent%2Fperformance-tuning.html%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Eperformance%20tuning%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ESome%20future%20improvements%20I%20might%20add%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EImplement%20impstats%20for%20rsyslog%20and%20send%20data%20to%20Log%20Analytics.%26nbsp%3B%20This%20would%20allow%20performance%20monitoring%20of%20rsyslog%20(dashboarding%2C%20queries)%3C%2FLI%3E%0A%3CLI%3EImplement%20GeoIP%20in%20rsyslog%3C%2FLI%3E%0A%3CLI%3EImplement%20Logstash%20monitoring%20APIs%20and%20send%20data%20to%20Log%20Analytics.%26nbsp%3B%20This%20would%20allow%20performance%20monitoring%20of%20Logstash%20(dashboarding%2C%20queries)%3C%2FLI%3E%0A%3CLI%3ECreate%20additional%20sample%20using%20fluentd%3C%2FLI%3E%0A%3CLI%3ECreate%20additional%20sample%20using%20syslog-ng%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EThanks%20for%20reading!%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1185854%22%20slang%3D%22en-US%22%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22CEF1.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F172375iEC8DB274FBECB6FB%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22CEF1.png%22%20alt%3D%22CEF1.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3ELearn%20how%20to%20scale%20up%20Syslog%20CEF%20collection%20in%20Azure%20Sentinel%20using%20Azure%20Virtual%20Machine%20Scale%20Sets%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1186188%22%20slang%3D%22en-US%22%3ERe%3A%20Scaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1186188%22%20slang%3D%22en-US%22%3E%3CP%3EI%20moved%20this%20feedback%20to%20the%20github%20repo%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1189072%22%20slang%3D%22en-US%22%3ERe%3A%20Scaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1189072%22%20slang%3D%22en-US%22%3EHi%20Nicholas%2C%20I%20was%20wondering%20if%20you%20know%20how%20I%20can%20daisy%20chain%20collectors%20like%20in%20the%20timed%20youtube%20link%20(%3CA%20href%3D%22https%3A%2F%2Fyoutu.be%2F_mm3GNwPBHU%3Flist%3DPLOhMGpMOPKRHPHCvzia3EE5OY5EkQRCuH%26amp%3Bt%3D896%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fyoutu.be%2F_mm3GNwPBHU%3Flist%3DPLOhMGpMOPKRHPHCvzia3EE5OY5EkQRCuH%26amp%3Bt%3D896%3C%2FA%3E)%20was%20presented%20by%20one%20of%20Microsoft's%20Sentinel%20guys%20called%20Ofer.%20I%20was%20able%20to%20install%20the%20CEF-Syslog%20Ubuntu%20server%20on-prem%20but%20I%20am%20trying%20to%20do%20the%20Syslog-Collector%20proxy.%20I%20believe%20there%20must%20be%20some%20configs%20that%20need%20to%20change%20on%20both%20the%20nodes.%20Simply%20the%20architecture%20I%20was%20to%20put%20is%20such%3A%201.%20Install%20a%20CEF-Syslog%20on-prem%20Ubuntu%20machine%20to%20collect%20all%20CEF%20and%20Syslog%20sources%202.%20Forward%20the%20data%20from%20the%20CEF-Syslog%20machine%20to%20another%20Syslog-Collector-Proxy%20in%20another%20segment%20of%20my%20network%20to%20forward%20to%20Azure%20to%20Sentinel%20Can%20you%20help%3F%20Thank%20you%2C%20Egal%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1535529%22%20slang%3D%22en-US%22%3ERe%3A%20Scaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1535529%22%20slang%3D%22en-US%22%3E%3CP%3EI%20see%26nbsp%3B%3CSPAN%3EStandard_B16ms%20sku%20is%20used%20in%20the%20vmss.%3C%2FSPAN%3E%3CBR%20%2F%3E%3C%2FP%3E%3CP%3E%3CSPAN%3EWas%20this%20a%20decision%20or%20just%20by%20sample%3F%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1536111%22%20slang%3D%22en-US%22%3ERe%3A%20Scaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1536111%22%20slang%3D%22en-US%22%3E%3CP%3EHi%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F602554%22%20target%3D%22_blank%22%3E%40eciruam%3C%2FA%3E%26nbsp%3BI%20actually%20need%20to%20update%20the%20template.%26nbsp%3B%20I%20worked%20with%20the%20agent%20team%20and%20they%20did%20some%20testing.%26nbsp%3B%20With%20an%26nbsp%3B%3CSTRONG%3EF4s_v2%20%3C%2FSTRONG%3EVM%20size%20they%20were%20able%20to%20achieve%20higher%20EPS%20and%20recommended%20that%20as%20the%20size.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1785297%22%20slang%3D%22en-US%22%3ERe%3A%20Scaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1785297%22%20slang%3D%22en-US%22%3E%3CP%3ENicholas%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Edo%20you%20have%20any%20concerns%20about%20scaling%20down%20a%20vm%20scale%20set%20without%20knowing%20if%20it's%20queue%20is%20empty%3F%26nbsp%3B%20isn't%20there%20some%20risk%20of%20lost%20messages%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20am%20toying%20with%20the%20idea%20of%20adapting%20my%20syslog-ng%20front-ends%20to%20push%20messages%20into%20a%20service%20bus%20queue%20(or%20multiple%20queues)%2C%20and%20then%20using%20a%20custom%20input%20into%20logstash%20to%20peek%20a%20message%20from%20the%20queue%2C%20process%20it%20through%20logstash%2C%20and%20then%20deque%20the%20message%20after%20processing.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThat%20would%20remove%20the%20need%20for%20load%20balancers%2C%20and%20allow%20the%20scale%20sets%20to%20expand%2Fcontract%20based%20on%20load%20(or%20ideally%20service%20bus%20queue%20sizes)%2C%20and%20guarantee%20delivery%20of%20the%20messages.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Ethoughts%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1785678%22%20slang%3D%22en-US%22%3ERe%3A%20Scaling%20Up%20Syslog%20CEF%20Collection%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1785678%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F11818%22%20target%3D%22_blank%22%3E%40Justin%20Ainsworth%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3Etheoretically%2C%20yea%20i%20guess%20that%20could%20happen.%26nbsp%3B%20but%20you%20could%20set%20autoscale%20settings%20to%20check%20network%20out%20instead%20of%20CPU.%26nbsp%3B%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fvirtual-machine-scale-sets%2Fvirtual-machine-scale-sets-autoscale-overview%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fvirtual-machine-scale-sets%2Fvirtual-machine-scale-sets-autoscale-overview%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3Eyea%20you%20could%20do%20some%20kind%20of%20bus%20in%20azure.%26nbsp%3B%20i%20went%20the%20simple%20route%20to%20solve%20a%20customer%20request.%3C%2FP%3E%3C%2FLINGO-BODY%3E

This blog post is authored by Nicholas DiCola

 

In the last few months working on Azure Sentinel, many customers have asked me about scaling up syslog CEF collection for getting data into Azure Sentinel.  I have created two sample architectures with code deployment for this purpose.  The samples is available at:

 

CEF-VMSS is for deploying native Azure Sentinel CEF collection by sending syslog CEF message to rsyslog which then sends the messages to the Log Analytics Agent.

 

Logstash-VMSS is for deploying Logstash on the VMs to do message manipulation which then sends the messages to the Log Analytics Agent.  You may also want to use this architecture and change the input to a source like Kafka.

 

I will not deep dive on all the topics of this architecture.  You can research each on your own and will focus on an overview of the architecture.

 

Virtual Machine Scale Set

The architecture starts with a VMSS which lets you manage and create a group of virtual machines.  These VMs can autoscale up and down additional instances based on schedule or demand.  The sample uses autoscale settings to configure the VMSS to scale up and down based on CPU (demand) of messages being sent.

 

I have included a Load Balancer in-front of the VMSS which will allow you to configure 1 destination IP address (the Public Ip Address) and it will spread the incoming messages across the running instances.

 

There are 2 ARM templates for RedHat and Unbuntu.  The templates deploy everything needed for the architecture.  One key part of the ARM templates is using cloud init to configure the VMSS instances as they are created.  Below is the Unbuntu cloud init files.

 

Cloud Init for CEF-VMSS:

#cloud-config
package_upgrade: true
runcmd:
  - sudo apt-get update
  - sudo wget https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/CEF/cef_installer.py&&sudo python cef_installer.py

As you can see the cloud init, it installs updates and the Log Analytics Agent using the Azure Sentinel CEF script.  The ARM template appends the workspace id and workspace key to the last line so that the agent gets connected to the right workspace

 

Cloud Init for Logstash-VMSS:

#cloud-config
package_upgrade: true
packages:
  - default-jre
runcmd:
  - wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
  - sudo apt-get install apt-transport-https
  - echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
  - sudo apt-get update
  - sudo apt-get install logstash
  - sudo /usr/share/logstash/bin/logstash-plugin install logstash-output-syslog
  - sudo /usr/share/logstash/bin/logstash-plugin update
  - wget -q https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Logstash-VMSS/logstash.config -O /etc/logstash/config.d/logstash.config
  - echo "update this line with wget -q https://sourceURL -O /etc/logstash/pipelines.yml if you have a custom pipelines file"
  - sudo systemctl start logstash.service
  - sudo wget https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/CEF/cef_installer.py&&sudo python cef_installer.py

It installs Java, Logstash, Logstash Syslog Output plugin and the Log Analytics Agent using the Azure Sentinel CEF script.  The ARM template appends the workspace id and workspace key to the last line so that the agent gets connected to the right workspace.

CEF

CEF is our default way to collect external solutions like firewalls and proxies.  The CEF install script will install the Log Analytics agent, configure rsyslog, and configure the agent for CEF collection.

Logstash

Logstash dynamically ingests, transforms, and ships your data regardless of format or complexity.  It has many input, filter and output plugins.  This can allow you to get data from many sources, manipulate the event data and output to the Log Analytics Agent locally on the machine.  There are many input plugins so this makes it easy to connect to other sources like Kafka.

 

Here is the sample logstash.conf file that is used in the sample architecture:

input {
  tcp {
    port => 5514
    type => syslog
    codec => cef
  }
  udp {
    port => 5514
    type => syslog
    codec => cef
  }
}

filter {
  geoip {
    source => "src"
    target => "srcGeoIP"
    add_field => { "sourceLongitude" => "%{[srcGeoIP][longitude]}" }
    add_field => { "sourceLatitude" => "%{[srcGeoIP][latitude]}" }
  }
  geoip{
    source => "dst"
    target => "dstGeoIP"
    add_field => { "destinationLongitude" => "%{[dstGeoIP][longitude]}" }
    add_field => { "destinationLatitude" => "%{[dstGeoIP][latitude]}" }
  }
  mutate{
    add_field => { "agentReceiptTime" => "%{@timestamp}" }
  }
}

output {
  syslog {
  host => "127.0.0.1"
  port => 25226
  protocol "tcp"
  codec => cef {
    reverse_mapping => true
    delimiter => "\r\n"
    vendor      => "%{deviceVendor}"
    product     => "%{deviceProduct}"
    version     => "%{deviceVersion}"
    signature   => "%{deviceEventClassId}"
    name        => "%{name}"
    severity    => "%{severity}"
    fields => [
      "deviceAction",
      "applicationProtocol",
      "deviceCustomIPv6Address1",
      "deviceCustomIPv6Address1Label",
      "deviceCustomIPv6Address2",
      "deviceCustomIPv6Address2Label",
      "deviceCustomIPv6Address3",
      "deviceCustomIPv6Address3Label",
      "deviceCustomIPv6Address4",
      "deviceCustomIPv6Address4Label",
      "deviceEventCategory",
      "deviceCustomFloatingPoint1",
      "deviceCustomFloatingPoint1Label",
      "deviceCustomFloatingPoint2",
      "deviceCustomFloatingPoint2Label",
      "deviceCustomFloatingPoint3",
      "deviceCustomFloatingPoint3Label",
      "deviceCustomFloatingPoint4",
      "deviceCustomFloatingPoint4Label",
      "deviceCustomNumber1",
      "deviceCustomNumber1Label",
      "deviceCustomNumber2",
      "deviceCustomNumber2Label",
      "deviceCustomNumber3",
      "deviceCustomNumber3Label",
      "baseEventCount",
      "deviceCustomString1",
      "deviceCustomString1Label",
      "deviceCustomString2",
      "deviceCustomString2Label",
      "deviceCustomString3",
      "deviceCustomString3Label",
      "deviceCustomString4",
      "deviceCustomString4Label",
      "deviceCustomString5",
      "deviceCustomString5Label",
      "deviceCustomString6",
      "deviceCustomString6Label",
      "destinationHostName",
      "destinationMacAddress",
      "destinationNtDomain",
      "destinationProcessId",
      "destinationUserPrivileges",
      "destinationProcessName",
      "destinationPort",
      "destinationAddress",
      "destinationUserId",
      "destinationUserName",
      "deviceAddress",
      "deviceHostName",
      "deviceProcessId",
      "endTime",
      "fileName",
      "fileSize",
      "bytesIn",
      "bytesOut",
      "eventOutcome",
      "transportProtocol",
      "requestUrl",
      "deviceReceiptTime",
      "sourceHostName",
      "sourceMacAddress",
      "sourceNtDomain",
      "sourceProcessId",
      "sourceUserPrivileges",
      "sourceProcessName",
     "sourcePort",
      "sourceAddress",
      "startTime",
      "sourceUserId",
      "sourceUserName",
      "agentHostName",
      "agentReceiptTime",
      "agentType",
      "agentId",
      "cefVersion",
      "agentAddress",
      "agentVersion",
      "agentTimeZone",
      "destinationTimeZone",
      "sourceLongitude",
      "sourceLatitude",
      "destinationLongitude",
      "destinationLatitude",
      "categoryDeviceType",
      "managerReceiptTime",
      "agentMacAddress"
      ]
    }
  }
}

The inputs accept both TCP and UDP on port 5514.  I used 5514 because Logstash runs as non-root and requires special configuration to use port 514.  I decided to keep it simple.  On input, its expecting CEF format using “codec => cef” and tags the event as syslog.

 

Once the event is accepted, I have added a few filters.  The first uses the GeoIP plugin which uses the local GeoLite2 database to lookup the source and destination IP addresses.  These are added to a custom field and to align with RFC compliance I then add_field to bring the latitude and longitude into proper fields.  I also mutate to copy the message received time into agentRecievedTime.  This is important as Logstash will send the message to the Log Analytics using its time which will end up as TimeGenerated in Log Analytics.  Doing this will allow you to see the original send time and the time Logstash sent it.  A simple compare will show you how long its take to process.

 

In the output section, I use the syslog plugin to output the message to the agent.  The agent listens on TCP 127.0.0.1:25226.  I set the output plugin to the CEF codec again and there are couple of key important configs.  “reverse_mapping => true” ensures that the message is sent using the short names (src vs sourceAddress) which is required by Log Analytics.  The fields portion requires all fields you want to send. I have included all fields the CEF codec supports.  If the fields doesn’t exist it wont be sent.

Azure Sentinel

Once the data is sent to the agent, it will follow all the normal CEF collection ingestion process and end up in CommonSecurityLog.  You can monitor the VMSS event per second using the following query:

CommonSecurityLog
| where _TimeReceived > ago(20m)
| summarize count() by bin(_TimeReceived, 1m), _ResourceId
| extend count =count_ /60
| sort by _TimeReceived desc

This will get all logs in the last 20m and summarize by TimeRecieved and ResourceId.  This gives you the # of event per minute, so you need to create a count column equal to count_ divided by 60 seconds.  Now you can see the EPS per VMSS instance.

If you have performance issues, I recommend you look into Ryslog performance or Logstash performance tuning.

 

Some future improvements I might add:

  • Implement impstats for rsyslog and send data to Log Analytics.  This would allow performance monitoring of rsyslog (dashboarding, queries)
  • Implement GeoIP in rsyslog
  • Implement Logstash monitoring APIs and send data to Log Analytics.  This would allow performance monitoring of Logstash (dashboarding, queries)
  • Create additional sample using fluentd
  • Create additional sample using syslog-ng

Thanks for reading!

6 Comments
Visitor

I moved this feedback to the github repo

Senior Member
Hi Nicholas, I was wondering if you know how I can daisy chain collectors like in the timed youtube link (https://youtu.be/_mm3GNwPBHU?list=PLOhMGpMOPKRHPHCvzia3EE5OY5EkQRCuH&t=896) was presented by one of Microsoft's Sentinel guys called Ofer. I was able to install the CEF-Syslog Ubuntu server on-prem but I am trying to do the Syslog-Collector proxy. I believe there must be some configs that need to change on both the nodes. Simply the architecture I was to put is such: 1. Install a CEF-Syslog on-prem Ubuntu machine to collect all CEF and Syslog sources 2. Forward the data from the CEF-Syslog machine to another Syslog-Collector-Proxy in another segment of my network to forward to Azure to Sentinel Can you help? Thank you, Egal
New Contributor

I see Standard_B16ms sku is used in the vmss.

Was this a decision or just by sample?

Hi @eciruam I actually need to update the template.  I worked with the agent team and they did some testing.  With an F4s_v2 VM size they were able to achieve higher EPS and recommended that as the size.

Senior Member

Nicholas,

 

do you have any concerns about scaling down a vm scale set without knowing if it's queue is empty?  isn't there some risk of lost messages?

 

I am toying with the idea of adapting my syslog-ng front-ends to push messages into a service bus queue (or multiple queues), and then using a custom input into logstash to peek a message from the queue, process it through logstash, and then deque the message after processing.

 

That would remove the need for load balancers, and allow the scale sets to expand/contract based on load (or ideally service bus queue sizes), and guarantee delivery of the messages.

 

thoughts?

@Justin Ainsworth 

theoretically, yea i guess that could happen.  but you could set autoscale settings to check network out instead of CPU.  https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-autosca...

 

yea you could do some kind of bus in azure.  i went the simple route to solve a customer request.